Redis worker stops consuming after deploy because BRPOP timeout 0 blocks graceful shutdown

2026-11-13 · 4 min read

Your 14:00 deploy sends SIGTERM to each worker. Ten minutes later, half the pods are still up, some have been SIGKILLed mid-job, retry logic has re-pushed the same payloads, and a few customers get the welcome email twice. The shutdown handler logged that it ran. The process still sat there, parked in a blocking Redis read, until the orchestrator ran out of patience.

This is the most common way a worker quietly breaks a deploy pipeline — and the fix is a one-character change at its core.

Why `timeout 0` blocks forever

BRPOP queue 0 means "no timeout." Per the Redis command docs, a zero timeout blocks the client indefinitely; the call returns when an element arrives, on error, or when the connection closes. [link needed]

The worker loop usually looks like this:


while running:
    _, payload = redis.brpop("jobs", 0)
    handle(payload)

The SIGTERM handler sets running = False. But the loop is not evaluating running — it is inside the client's blocking socket read. Nothing re-checks the flag until a job arrives. On a quiet queue, that may be never. In Python, signal handlers run in the main thread, and the underlying system call may even be restarted after the handler returns. [link needed] So the flag flips, the process keeps blocking, and the patience of your orchestrator becomes the real timeout.

Note what this costs: the grace period is spent entirely on waiting, so every deploy pins pods (and memory, and connections) for the full 30 seconds, and any job longer than the grace period is killed mid-flight.

Make Redis BRPOP graceful shutdown deterministic

The mechanism you want: a short, finite block, with the shutdown flag checked between blocks.

1. Replace 0 with a small finite timeout — 1 to 4 seconds. Older Redis versions accept integer timeouts only; fractional values need Redis 6.0 or later, so check your server version. [link needed]

2. Install a SIGTERM handler that sets the flag and does nothing else. Do not call exit() inside it.

3. Check the flag at the top of the loop and break when it is set.

4. After the break, finish the in-flight job, then ack and clean up. If you use BRPOPLPUSH/BLMOVE to a processing list, remove the completed job from that list before exiting.

5. Set the termination grace period above worst-case job duration *plus* one timeout, or enforce a hard per-job deadline.

6. Keep handlers idempotent anyway, so a SIGKILL at step 4 is survivable.

The trade-off is real but small: each idle worker wakes every few seconds instead of sleeping indefinitely, so expect roughly one extra round trip per timeout window and up to one timeout of extra exit latency. That is a better deal than an unbounded exit.

Test it on a quiet queue before you trust it

Deploy the change to staging and deliberately leave the queue empty. Send SIGTERM to one worker by hand and time the exit; expect the process to finish within one timeout plus the current job. Confirm the shutdown log line appears before the exit. Then run it under load, with a job that takes 80% of the grace period, and watch for redelivery. If you cannot reproduce a clean exit on an empty queue, you have not fixed anything — a busy queue hides the bug, because an arriving job wakes the loop and lets the flag be read by accident.

When NOT to do this

If your jobs are sub-second, idempotent, and your supervisor kills after one second anyway, a timeout adds polling load and a code path that never gets exercised — accept the kill. The same applies if you already use a blocking read on Redis Streams (XREADGROUP ... BLOCK), which has its own semantics; don't copy this pattern over without re-reading those docs. And if a worker's job is to drain a large backlog before exiting, a 2-second timeout is not the lever — the drain policy is.

Checklist

[ ] No BRPOP/BLPOP call with timeout 0 in the request path
[ ] Finite timeout (1–4s) chosen and committed
[ ] SIGTERM handler sets a flag only
[ ] Flag checked every loop iteration
[ ] In-flight job finished and acked after break
[ ] Grace period > max job duration + timeout
[ ] Verified with a manual SIGTERM on an empty queue in staging