✓Draft ADR-191 for billing-webhook v2 migrationDiscussed in Tuesday standup · attached link in slack@r.lansingdue 20 NOVDONE
ACTION ITEMS · MEETING NOTES14 / 24
CHANGELOG · W47v2.4.18 → v2.5.0
CHANGELOG · WEEK 47
What we shipped, what we changed, what we broke.
★ ADDEDWebhook replay button on the billing dashboardOperators can now retry any failed delivery directly from the UI. Replay rate-limited at 1/sec per workspace.
v2.5.0PR · #4218@r.lansing
CHANGELOG · NOTES.OPS.INTERNAL15 / 24
ROLLOUT · v2.5.0 · DAY 4 OF 7NEXT GATE: 12:00 UTC
ROLLOUT · v2.5.0 · STAGED
Currently rolling out to 25% of traffic.
★ COMPLETE
Canary · 0.1%
internal team, 4 hours, 0 errors
★ COMPLETE
Wave 1 · 5%
24 hours, 1 minor warning
▶ IN PROGRESS
Wave 2 · 25%
2 hours in, p95 stable
PENDING
Wave 3 · 100%
scheduled · 26 NOV 09:00
ROLLOUT · v2.5.0 · STAGED ROLLOUT16 / 24
ON-CALL · NOV 2026generated by oncall-bot
ON-CALL ROTATION · NOV
Four weeks, four pairs.
WEEK 453—9 NOV
★ PRIMARY
P. Adelmann
SECONDARY
H. Reuter
2 incidents this week
ON-CALL · OPSGENIE · ROTATION-OPS-PRIMARY17 / 24
QUERY · 22 NOV 18:42184 rows · 0.42s
QUERY RESULT · BILLING DB · READ-ONLY REPLICA
Failed webhooks in the last 24 hours.
-- find failed webhook deliveries, last 24h SELECT id, workspace_id, attempted_at, http_status, error_class FROM webhook_deliveries WHERE succeeded = false AND attempted_at > now() - interval '24 hours' ORDER BY attempted_at DESC LIMIT184;
ID
WORKSPACE
ATTEMPTED AT
STATUS
ERROR
whd_184a02f
ws_4218
18:42:18 UTC
503
UpstreamTimeoutError
QUERY · BILLING-DB-READ · NON-INDEXED18 / 24
VENDORS · 12 ACTIVE€18,420 / mo
VENDOR LIST · EXTERNAL DEPENDENCIES
Twelve vendors. Two up for renewal.
VENDOR
USED FOR
€/mo
RENEWS
OWNER
SLA
Datadog
observability, metrics
€4,820
15 MAR 2027
@p.adelmann
★ 99.9
VENDORS · CONFLUENCE/OPS/VENDORS19 / 24
NOTES · 22 NOV · WEEKLY OPSv1 · m.kovaleva
MEETING NOTES · WEEKLY OPS · 22 NOV
Weekly ops sync · forty-two minutes.
WHEN
Tue 22 NOV · 14:00 — 14:42 UTC
WHERE
Zoom · #ops-weekly
ATTENDEES
★ M. Kovaleva (chair), R. Lansing, P. Adelmann, H. Reuter, M. Halász
FACILITATOR
M. Kovaleva
★ 01Billing-webhook migration · readiness
ADR-191 accepted. Staged rollout already at 25%, no errors above baseline. Decision: continue to 100% on Friday, with M. Halász on standby for support escalations.