Webhooks 2.5.0 uses one durable delivery model that separates broker ingestion from outbound HTTP attempts. This page defines what the pipeline guarantees, where duplicates can occur, how retries are scheduled, and how operators inspect and replay deliveries.
The worker does not expose a runtime pipeline selector. Delivery inspection and replay APIs use the same persisted model as automatic dispatch.
Upgrade from the attempts model#
New installations start directly with durable deliveries. Existing installations that still have outstanding retries in the former attempts table require a coordinated upgrade:
- Stop every old Webhooks worker and wait for it to terminate.
- Apply the new database migrations.
- Run
webhooks backfill-deliveriesuntil it completes. - Deploy the new Webhooks version and recreate its workers.
The backfill command is resumable and idempotent. It exists only as an upgrade adapter: the runtime does not read or write the old attempts queue, and running old and new workers together is not supported. Operator-managed installations coordinate this sequence during the upgrade.
Delivery pipeline#
For each broker event, Webhooks selects active configurations that subscribe to its exact event type. In one database transaction, it creates one pending delivery per matching configuration. The pair (event ID, configuration ID) is unique, so broker redelivery cannot enqueue the same delivery twice. Webhooks acknowledges the broker message only after this transaction commits.
The dispatcher then claims due rows with database locks, marks them delivering, and performs outbound requests independently of broker consumption. Multiple workers can claim work without intentionally selecting the same row.
Durable guarantee summary#
| Property | Contract |
|---|---|
| Acceptance | A matching delivery is stored before the broker event is acknowledged. If persistence fails, the broker can redeliver the event. |
| Broker deduplication | Repeated broker delivery of the same event does not create another delivery for the same configuration. |
| HTTP delivery | At-least-once attempts until the endpoint returns 2xx or the delivery reaches a terminal condition. Successful processing by the receiver is not guaranteed. |
| Duplicates | Possible. A request can reach the endpoint before Webhooks loses the response or fails to commit the attempt result. Receivers must deduplicate. |
| Ordering | Not guaranteed across events or configurations. The dispatcher is concurrent and retries can overtake earlier deliveries. |
| Timeout | Each HTTP attempt has a 30-second timeout. |
| Retry budget | At most 15 attempts and at most 10 hours per retry generation with default settings. The first limit reached terminates the generation. |
| Manual recovery | Failed deliveries can receive a fresh retry generation through the replay API. Pending deliveries can be expedited. |
This is not an exactly-once protocol. Exactly-once business effects require idempotency in the receiving application.
Durable delivery states#
| Status | Meaning |
|---|---|
pending | Stored and waiting for its first or next attempt. nextAttemptAt indicates when it becomes eligible. |
delivering | Claimed by a dispatcher worker and currently in flight. |
succeeded | The endpoint returned a 2xx response. This is terminal. |
failed | The endpoint returned a permanent error, or the retry count or elapsed retry window was exhausted. |
cancelled | Delivery stopped because its configuration was deactivated or deleted. |
Every HTTP call creates an append-only attempt record with the endpoint, attempt number, replay generation, outcome, status code, duration, response excerpt, error, and timestamp. If a worker stops while a delivery is delivering, claims older than five minutes are recovered: active configurations return to pending; inactive or deleted configurations become cancelled.
Response classification#
Webhooks classifies the final response from the endpoint as follows:
| Result | Delivery action |
|---|---|
2xx | Mark succeeded; no more automatic attempts. |
408 Request Timeout | Retry. |
429 Too Many Requests | Retry and honor a valid Retry-After when it requests a longer delay. |
Other 4xx | Mark failed immediately. These errors normally require a configuration or application change. |
5xx | Retry. A valid Retry-After can extend the delay. |
| Network error or 30-second timeout | Retry. The attempt records status code 0. |
The response body does not affect classification. Webhooks reads at most 64 KiB of it for diagnostics.
Return 429 with Retry-After when your service is healthy but temporarily rate-limited. Return another 4xx only when retrying the same request cannot succeed without a change, because Webhooks treats it as terminal.
Retry schedule#
The default policy uses exponential backoff without jitter. The first request is immediate, then delays double from one minute until they reach the one-hour cap:
| Attempt | Approximate time from first attempt |
|---|---|
| 1 | Immediately |
| 2 | 1 minute |
| 3 | 3 minutes |
| 4 | 7 minutes |
| 5 | 15 minutes |
| 6 | 31 minutes |
| 7 | 1 hour 3 minutes |
| 8 | 2 hours 3 minutes |
| 9–15 | Once per hour, with attempt 15 at approximately 9 hours 3 minutes |
Automatic retries stop when either limit is reached:
- 15 total attempts in the current generation;
- 10 elapsed hours from the first attempt in the current generation.
For retryable responses, Webhooks uses Retry-After when it is valid and longer than the computed backoff. It accepts delay-seconds or an HTTP date, caps an endpoint-supplied delay at six hours, and never allows it to extend the 10-hour retry window.
Webhooks does not open a per-endpoint circuit breaker or automatically disable a configuration after repeated failures. Attempt and time caps bound each delivery; operators must monitor terminal failures and queue growth.
Configuration changes during delivery#
Deactivating or deleting a configuration cancels its pending deliveries. An in-flight attempt can finish, but any non-successful result becomes cancelled instead of returning to the queue. Reactivating a configuration does not resurrect cancelled deliveries; replay also rejects cancelled deliveries.
Updating an endpoint affects later attempts because the dispatcher reads the current configuration before sending. Rotating the secret similarly causes later attempts to use the new secret.
Inspect deliveries#
List deliveries by configuration, status, or creation window. The list excludes payloads and uses an opaque cursor; request one delivery to retrieve its payload.
Valid statuses are pending, delivering, succeeded, failed, and cancelled. pageSize defaults to 100 and cannot exceed 1,000.
Retrieve a delivery and its attempt history:
Use lastStatusCode, lastError, nextAttemptAt, and the attempts list to distinguish an endpoint rejection from a timeout, retry backlog, or exhausted retry budget.
Replay one delivery#
Only failed and pending deliveries belonging to an active configuration are eligible for replay. A replay request requires an Idempotency-Key header.
For a failed delivery, replay increments replayGeneration, resets its attempt count and 10-hour retry window, and queues it immediately. For a pending delivery, replay only moves the next attempt to now; it does not reset the current budget. succeeded, delivering, and cancelled deliveries are not eligible for replay.
The same idempotency key and request return the original replay result for 24 hours. Reusing the key for a different replay returns a conflict.
Replay a bounded set#
Bulk replay operates synchronously on one page of at most 1,000 deliveries. A request must include createdAtFrom; its creation-time window must be positive and cannot exceed 90 days.
The response separates replayed failed deliveries, expedited pending deliveries, and skipped rows. When hasMore is true, submit another request with nextCursor and a new Idempotency-Key. Keep the original filters unchanged; the cursor binds to the time window, statuses, and configuration IDs.
Retention and observability#
With default settings, Webhooks retains succeeded deliveries for 30 days and failed or cancelled deliveries for 90 days. Attempt history is deleted with its delivery. Retention runs hourly. These values are runtime settings for self-hosted deployments and can be changed or disabled.
Webhooks exports OpenTelemetry traces and the following delivery metrics when metrics export is configured:
| Metric | Purpose |
|---|---|
webhooks_delivery_attempts_total | Attempts by outcome and HTTP status class. |
webhooks_delivery_duration_seconds | Outbound request duration. |
webhooks_retry_queue_depth | Pending delivery count, capped at 1,000,000. |
webhooks_replayed_deliveries_total | Deliveries replayed or expedited manually. |
webhooks_delivery_transitions_total | Durable delivery state transitions. |
webhooks_delivery_claims_recovered_total | Stale in-flight claims recovered after worker interruption. |
Alert on a growing retry queue, sustained 5xx or timeout rates, terminal failures, and old pending deliveries. A delivery can be safely considered complete only when it is succeeded, or when your operational process has accepted its terminal failure.