Handling webhook retries without processing the same event twice

2026-11-09 · 3 min read

Handling webhook retries without processing the same event twice

At 2:03 a.m., your handler charges a customer $49, then the load balancer times out before your 200 reaches the sender. The sender retries. Your handler charges again. The fix is not a longer timeout; it is treating each delivery as a possible duplicate and using a stable identity for the event.

Use a webhook idempotency key at the boundary

Many webhook senders retry on timeouts and non-2xx responses [link needed]. Assume every delivery can arrive more than once, out of order, or after a crash.

A webhook idempotency key is the event’s stable identity, usually the provider’s event ID. Store it with your tenant/account ID and event type. If the provider has no event ID, derive a deterministic key from stable fields such as source, event type, resource ID, and a normalized body digest. Do not generate a random key per HTTP request. Also do not use a payload hash if the sender adds volatile delivery timestamps.

Enforce uniqueness in storage, not in application memory. A relational unique constraint on (source, tenant_id, idempotency_key) is simple and checkable.

Claim the key before side effects

Use this procedure:

1. Verify the webhook signature and reject invalid requests.

2. Extract the webhook idempotency key. If it is missing and cannot be derived safely, return a non-2xx response so the sender retries or alerts.

3. Insert a row with status processing and let the unique constraint decide the race. If the insert succeeds, you own this event.

4. If the insert conflicts, read the existing row. If status is done, return 2xx. If status is processing, either return 2xx for async processing or wait briefly; reclaim rows stuck in processing after a timeout.

5. Perform side effects. Where downstream APIs support idempotency keys, pass a deterministic key derived from your webhook idempotency key [link needed].

6. Mark the row done only after durable state is written. On failure, mark failed and return 5xx so the sender retries.

7. For slow work, write the event and an outbox record in one transaction, return 2xx, and let a worker process it. The worker uses the same key.

Your webhook idempotency key should be stored before charges, emails, provisioning, or queue publishes. If you store it after side effects, a crash in between leaves no record and the retry repeats the work.

Make retries safe after crashes

A key alone does not make external systems transactional. You can still get this sequence: insert key, charge card, crash before marking done. On retry, the key is found but the local status is stale.

Handle that with a stale-processing timeout and a reclaim step. Before reclaiming, check downstream state where possible: does the order already have the charge ID? Is the email already queued? For writes to APIs that support idempotency keys, reuse the same downstream key so the second call returns the first result instead of creating a second resource [link needed].

Record enough state to resume: event ID, tenant, status, attempt count, last error, and timestamps. Keep logs searchable by key.

When NOT to do this

Do not add a webhook idempotency key table if the sender already guarantees unique delivery and your side effects are fully transactional with the event record. Do not use one global key across event types; the same event ID from two sources can collide. Do not deduplicate only by resource ID: one order can legitimately produce order.updated several times. And do not return 2xx before the key is durably stored.

Checklist

ad slot — paste the network tag here