Webhook delivery

The 2xx-to-ack contract

Return any 2xx status to acknowledge a delivery. As soon as we see a 2xx, the event is done — we will not send it again. Any non-2xx response, a connection error, or a timeout counts as a failed delivery and schedules a retry (see the schedule below).

Acknowledge before you do the work. The body of a webhook is intentionally minimal — the envelope plus a small data.object with the resource id and a few routing keys. Persist the event, return 200 immediately, then process it in a background job. Doing slow work (a follow-up GET, a downstream API call, heavy computation) inside the request risks the 30-second timeout, which we treat as a failure and retry — even though your handler may have finished the work after we gave up.

Request timeout: 30 seconds

Each delivery attempt gives your endpoint 30 seconds to return a response. If we do not receive one in that window, the attempt is aborted and recorded as a failure. This is a per-attempt budget, not a total budget — a slow response burns one retry and the next attempt starts the 30-second clock fresh.

Thirty seconds is a ceiling, not a target. Aim to respond in well under a second by acknowledging first and processing asynchronously. If your handler routinely approaches the timeout you will see spurious retries (and duplicate deliveries) whenever a request runs a little long.

Retry schedule

Failed deliveries are retried with exponential backoff plus jitter. The schedule is: base delay 1 s, multiplier ×2 per attempt, capped at 900 s (15 min) per hop, with ±20% jitter applied to each delay to avoid a thundering herd of redeliveries after an outage. Each event gets 1 initial delivery attempt, then up to 12 retries — 13 delivery attempts in total — before the event is exhausted.

After the initial attempt fails, each retry waits the following nominal (pre-jitter) delay before firing:

That is a cumulative retry window of roughly 47 minutes from the first attempt to exhaustion — sized deliberately to absorb a multi-minute receiver outage (for example, a rolling restart of your service) without dropping the event. Actual timing varies by up to ±20% per hop because of jitter.

After the final attempt fails, the event is marked exhausted and is not delivered again. Treat the retry window as your grace period for transient failures, not as a durable queue: if your endpoint is down longer than the window, use the bill sync endpoint to reconcile state you may have missed rather than expecting redelivery.

At-least-once, no ordering

Delivery is at-least-once. The retry mechanism above means the same event can arrive more than once: a network blip after your endpoint already committed, a response we never saw, or an internal redelivery on our side can all cause a duplicate. Your handler must be safe to run twice for the same event.

There is no ordering guarantee. Do not assume webhooks arrive in the order the underlying changes happened. Independent retries, backoff, and parallel delivery mean a bill.updated can land before the bill.created for the same resource. Drive your projection off the resource state you fetch (or off the envelope’s created timestamp for last-writer-wins), never off arrival order. If you need a strict, ordered view of a resource, reconcile with a GET or with bill sync.

Idempotent consumption: dedupe by event.id

Because delivery is at-least-once, idempotent consumption is not optional. Every webhook envelope carries a unique id (prefixed evt_) — use it as your dedupe key. The same id is also sent in the X-Webhook-Id header. Dedupe on event.id, not on request.idempotency_key (that field echoes the originating client’s POST key, when there was one, and is for tracing).

The handler below is the whole delivery contract in one place: it acknowledges fast, dedupes by event id, and records the id atomically with its side effect so a crash between the two cannot leak a duplicate. For the full pattern — TTLs, storage choices, and the outbound Idempotency-Key side — see Idempotency.

// Express-style handler. Signature verification (see the Webhooks guide)
// runs first; here we focus on ack + dedupe.
async function handleWebhook(req, res) {
  const envelope = req.body; // already signature-verified

  // 1) Have we processed this event id before? If so, ack and stop.
  if (await store.has(`webhook:${envelope.id}`)) {
    return res.status(200).end(); // idempotent replay — never double-process
  }

  // 2) Acknowledge FAST. Enqueue the real work; return 200 well under 30s.
  await queue.enqueue({ eventId: envelope.id, type: envelope.type, envelope });
  res.status(200).end();
}

// Worker: records the id atomically with the side effect so a crash
// between them can't leak a duplicate. Runs outside the 30s request window.
async function processEvent({ eventId, envelope }) {
  await store.transaction(async (tx) => {
    if (await tx.has(`webhook:${eventId}`)) return; // won the race already
    await tx.put(`webhook:${eventId}`, { ts: Date.now() }, { ttl: 7 * 86400 });
    await applyProjection(tx, envelope);
  });
}

404 on the follow-up GET is normal

Because delivery is asynchronous and some resources are short-lived, the GET /v1/<resource>/<id> you make after receiving an event can return 404 Not Found. This is expected: gone is gone. Log the event id and return 200 OK from your webhook endpoint — do not return non-2xx, or you will burn the retry budget re-fetching a resource that will still be gone on the next attempt.

The full rationale, and a handler that tolerates the 404, lives with the error docs: Error Handling — 404 after a webhook is normal.

Related

• Set Up Webhooks — register an endpoint, verify signatures, and the full event reference (catalog)
• API Reference: Webhooks — per-event payloads and headers
• Idempotency — inbound dedupe by event.id and outbound Idempotency-Key
• Error Handling — error envelope, retry behavior, and the 404-after-webhook rule