Webhook handler times out because it calls a slow API before returning 200
Webhook handler times out because it calls a slow API before returning 200
A fulfillment webhook started timing out at 3 a.m. The handler verified the signature, then called an internal shipping API to create the label, and only then returned 200. When that API's p99 climbed from 400 ms to 11 seconds, the provider's 10-second window [link needed] expired first. The provider retried. Each retry ran the whole handler again, so labels were created two and three times, the retry queue grew, and by morning support had 300 duplicate-shipment tickets plus a backlog that took four hours to drain.
None of that was a webhook problem. It was a response-ordering problem: the handler made the sender wait for work the sender doesn't care about.
How a webhook timeout slow API before response failure compounds
The first timeout is cheap. The damage comes from the retries:
- Duplicate side effects. Retries re-execute everything downstream of the verification step, including writes.
- Retry storms. Most providers retry with backoff and eventually give up [link needed], so a 10-minute downstream incident becomes an hour of redeliveries.
- Worker starvation. Each hung handler holds a connection and a process slot. A slow third-party API takes down handlers for unrelated event types on the same worker pool.
- Lost visibility. Timeouts look like "provider is flaky" in dashboards, so the actual dependency never gets investigated.
The fix: acknowledge first, process after the response
The synchronous part of a handler should do only what the sender needs: verify the signature, parse the payload, persist the raw body, enqueue a job, return 2xx. Everything else — enrichment, CRM writes, label creation, LLM calls — moves to a worker.
verify signature -> insert raw event (unique on provider event id) -> 200
|
v
worker: idempotent process + retry
The unique constraint on the provider's event ID is the part that makes retries safe. Without it, you've moved the duplication problem into the queue instead of solving it.
Procedure for migrating an existing handler
1. Measure first. Log wall-clock duration and the outbound calls in each handler for a week. You need the p99 to know which calls must leave the request path.
2. Classify calls. Keep anything the sender's semantics require (signature check, cheap validation of required fields). Move anything that mutates external state.
3. Add the raw event table with a unique index on event ID, plus a status column: received, processed, failed, dead.
4. Rewrite the handler to insert-or-ignore the raw event and return 200 in the same transaction. Return 4xx only for unparseable or unsigned payloads.
5. Build the worker with a queue consumer, idempotency check, exponential backoff, and a dead-letter destination. Cap retries at a number you can reason about.
6. Replay historical events in a shadow queue before cutover, and compare side effects against the synchronous version.
Expect the handler's p99 to drop to the cost of one insert. That is the entire goal.
When NOT to do this
Async isn't always correct:
- The provider needs a synchronous verdict. Some integrations expect a response body that decides what happens next [link needed]. You can't return
200and negotiate later. - Downstream is genuinely fast and bounded. An API with a 150 ms p99 and a hard timeout doesn't justify a queue, worker, and DLQ you now have to operate.
- The event is low-volume and unrepeatable. Ten events a day with no duplicate risk can tolerate a blocking call.
- You have no queue infrastructure and no one to run it. A database-backed job table is a valid middle ground; Redis plus a bespoke worker is not free.
Also skip this if the downstream call is the *only* reason the webhook exists — for example, a moderation gate whose result you must report back.
Verification metrics
Track handler p99 against the provider's timeout window, dead-letter depth, duplicate-detection hits, and queue age. Alert on the first and the last. If handler p99 approaches half the provider window, you're one incident away from the original failure.
Checklist
- [ ] Raw event persisted with a unique constraint on provider event ID
- [ ] Handler returns
2xxbefore any external mutation - [ ] Downstream calls moved to a worker with backoff and a retry cap
- [ ] Dead-letter queue exists and has an owner
- [ ] Duplicate attempts are logged, not silently swallowed
- [ ] Handler p99 and DLQ depth are on a dashboard with alerts
- [ ] Signature verification still happens synchronously, before the insert