Writing a minimal retry policy that does not duplicate payments: the failure cases to plan for
Writing a minimal retry policy that does not duplicate payments: the failure cases to plan for
At 02:14, a charge request times out after the provider has already captured the money. The client retries. The customer sees two charges, support issues a refund, and reconciliation takes an hour. An idempotent retry policy makes that second attempt return the first result instead of creating a second payment.
Retries are unavoidable on networks. The dangerous part is not the retry itself; it is retrying an operation whose first result is unknown. Plan for these cases before adding retry logic.
The failure cases to enumerate first
- Timeout after commit. The provider committed, but the response was lost. A blind retry charges again.
- Client crash between send and save. The request left the process, but the idempotency key or state was never persisted.
- Retry before the first response. A load balancer or SDK retries while the first request is still in flight.
- Provider 5xx after side effect. A 500 can mean "failed" or "succeeded, response failed" [link needed].
- Duplicate webhooks or events. Delivery retries can reapply a state change.
- Non-idempotent endpoints. POST is not idempotent by default [link needed]; not every payment API supports idempotency keys [link needed].
Write down which of these your client, SDK, load balancer, and provider can cause. Each one needs a different control.
Build an idempotent retry policy in five steps
1. Create the key before the first send. Generate a stable idempotency key per logical payment, not per HTTP attempt. Persist {key, amount, currency, customer, state=pending} before calling the provider.
2. Send the key and a request hash. Put the key in the provider's documented header or body field [link needed]. Store a hash of the canonical request so reused keys with different payloads are rejected.
3. Retry only eligible failures. Retry connection resets, timeouts, 429, and selected 5xx. Do not retry 400/422 unless the payload changed. Cap attempts and use exponential backoff with jitter [link needed].
4. Reuse the key on every attempt. Never generate a new key for the same logical payment. On success, store the provider response with the key. On a duplicate response, return the stored result.
5. Reconcile unknown states before acting. If the key state is pending and the provider supports lookup by key, query it. If not, wait for settlement or webhook before refunding or retrying [link needed].
Validate and observe
Your idempotent retry policy is only as good as the failure cases you test. Kill the client after send, delay the response past timeout, and replay the same key with a changed amount. Log key, attempt number, request hash, and provider reference. Alert on payments stuck pending beyond a threshold. Dashboard duplicate-key responses separately from fresh successes.
When NOT to do this
Do not add retries to an endpoint that has no idempotency support and no read-back lookup; an idempotent retry policy cannot make an unsafe operation safe by itself. Skip automatic retries for validation errors and for operations where a human must approve the new amount. If the provider's duplicate behavior is undocumented, keep retries manual until you can test it.
Checklist:
- [ ] Key generated before first send and persisted
- [ ] Request hash stored and checked on reuse
- [ ] Retry set limited to timeouts, resets, 429, selected 5xx
- [ ] Backoff capped with jitter
- [ ] Success response stored by key
- [ ] Pending payments reconciled before refund/retry
- [ ] Duplicate-key responses logged and alerted