Nginx keeps proxying to old pod IPs after rollout because upstream DNS is resolved only at startup

2026-11-14 · 3 min read

Nginx keeps proxying to old pod IPs after rollout because upstream DNS is resolved only at startup

At 03:14 a rolling update of checkout-api completed. All pods were Ready, the Service endpoints were updated, and Kubernetes had already terminated the old replicas. Nginx kept sending traffic to the three pod IPs it had learned hours earlier. Every request to a dead IP burned a proxy_connect_timeout before retrying, and users saw 502s for four minutes until someone ran nginx -s reload by hand.

That reload is the tell. Configuration load is the only moment nginx re-reads those hostnames.

The name is resolved once, at config load

When proxy_pass contains a literal hostname, or an upstream block lists a server hostname, nginx resolves it at startup and at every reload, then caches the result for the life of that configuration [link needed]. Nothing in the request path re-resolves. A failed connect retries the same cached set, so a rollout leaves you pinned to addresses that no longer exist until the next config change.

Kubernetes makes this worse: the DNS name you wrote is stable, but the A records behind it churn on every deploy.

Use the Nginx resolver upstream DNS path for runtime lookups

The fix is to force resolution out of config load and into the request path. Note that resolver alone is not enough — nginx only performs runtime lookups when the proxy target contains a variable.

1. Point nginx at your cluster DNS resolver with a short TTL, in the http or server block: resolver kube-dns.kube-system.svc.cluster.local valid=5s ipv6=off; [link needed]

2. Move the target into a variable named in the server block: set $backend "http://checkout-api.payments.svc.cluster.local:8080";

3. Change the directive to use it: proxy_pass $backend;. If you rely on URI rewriting, build the full URI in the variable instead (proxy_pass $backend$request_uri;).

4. Keep valid= short but non-zero. Values below a few seconds multiply DNS queries; 5–10s is a common starting point and should be tuned to your rollout duration.

5. Test and reload — nginx -t && nginx -s reload — never restart under load.

6. Verify the resolver address is reachable from inside the pod before you trust the reload: dig @<resolver-ip> checkout-api.payments.svc.cluster.local.

7. If you run open-source nginx 1.27.3 or newer, or NGINX Plus, the resolve parameter on an upstream server line is the better option: it re-resolves without giving up upstream blocks [link needed].

Be aware of the tradeoff — a variable in proxy_pass disables upstream-block features, including keepalive connection pooling and upstream-level load-balancing methods [link needed]. If you need those, prefer step 7.

Verifying it actually re-resolves

Add $upstream_addr to your log format, then scale the deployment up and down while tailing the log. New pod IPs should appear within roughly one valid= interval. As a stronger test, set resolver to a wrong IP and reload: if requests start failing immediately rather than at the next pod churn, resolution is happening at request time.

When NOT to do this

Checklist