Postgres hits max_connections during a rolling deploy because old pods keep pooled connections open

2026-11-14 · 3 min read

Postgres hits max_connections during a rolling deploy because old pods keep pooled connections open

Your deploy stalls at 60% with pods in CrashLoopBackOff and the app logs filling with FATAL: sorry, too many clients already. Postgres is at max_connections, but traffic hasn't gone up — the deploy itself did it. Each new pod opens its full pool the moment the process starts, while every old pod sits in Terminating for the full grace period still holding its pool open. For a few minutes you're running at roughly double capacity, and Postgres rejects the surplus. New pods fail readiness, the rollout never converges, and you're left rolling back at 2 a.m.

Why a postgres max_connections exhausted rolling deploy is arithmetic, not bad luck

Postgres allows max_connections minus superuser_reserved_connections slots for normal clients, and refuses the rest [link needed]. Your steady-state usage is fine because it's pods × pool_max. A rolling deploy temporarily makes that (old pods + new pods) × pool_max. If steady state runs at 70% of the limit, a 1.0 surge factor is enough to cross it.

The failure is usually compounded by shutdown behavior: runtimes don't close the pool on SIGTERM unless you tell them to, and many apps ignore SIGTERM entirely until Kubernetes sends SIGKILL. Old pods hold connections until the grace period expires.

Measure before you change anything

Run this against the primary and keep the output:


SELECT usename, application_name, client_addr, state, count(*)
FROM pg_stat_activity
GROUP BY 1,2,3,4 ORDER BY 5 DESC;

SELECT count(*), current_setting('max_connections')::int FROM pg_stat_activity;

Note which application_name values map to pods and whether the idle connections are in a pooler or in the app. If you run PgBouncer, the client_addr will be the pooler and the connection math lives there instead — check SHOW POOLS for cl_active and sv_active.

A procedure that actually fixes it

1. Compute the per-pod budget. pool_max = floor((max_connections - superuser_reserved - admin_headroom) / max_pods_during_deploy). Include surge pods and any batch jobs. Document the number in the repo.

2. Set the pool limit explicitly. Most pools default far too high per instance. Set max/maximumPoolSize to the computed value and cap idleTimeout so unused slots return.

3. Make shutdown drain. Add a preStop hook with a short sleep, handle SIGTERM in the app, stop accepting requests, close the pool, then exit. Set terminationGracePeriodSeconds above the drain time.

4. Serialize the rollout. Set maxSurge: 0 with maxUnavailable: 1 so old pods exit before new ones open pools, or use a partitioned/ordered rollout. It's slower and boring, which is the point.

5. Add or tune a pooler. PgBouncer in transaction mode collapses hundreds of app connections into a few dozen backend connections. A postgres max_connections exhausted rolling deploy largely disappears here, because the deploy only churns frontend connections.

6. Alert on the trend, not the wall. Page at 80% of max_connections sustained for five minutes, so the next deploy isn't the first time you find out.

Verify, don't assume

Re-run the deploy in staging with the same pod count and watch pg_stat_activity minute by minute. The connection count should stay flat and then dip as old pods drain. If it steps up, the preStop hook or signal handler isn't working — check kubectl describe pod for the actual termination timeline.

When NOT to do this

Don't reach for a transaction-mode pooler if your app depends on session state: LISTEN/NOTIFY, session-level advisory locks, SET without LOCAL, temp tables across statements, or prepared statements without driver support. Fix the drain and the pool math first. Also skip the pooler if you already run one in session mode for a small, well-headroomed fleet — adding a second layer just moves the math somewhere you'll forget to look.

Checklist