Nginx returns 502 after deploy because upstream DNS is cached at startup

2026-11-13 · 3 min read

Nginx returns 502 after deploy because upstream DNS is cached at startup

Your rolling deploy finishes green. Thirty seconds later the 502 rate graph goes vertical, the alert fires, and you're staring at error logs showing connect() failed (111: Connection refused) to an IP address that belonged to a container the orchestrator killed two minutes ago. Nothing in the nginx config changed. Nothing in the app changed. The only thing that happened is that DNS for the service name now points somewhere else — and nginx never asked again.

That's the failure mode behind every "it broke right after deploy and fixed itself when we restarted the proxy" incident. If nginx resolves an upstream hostname at config load, the address it got is the address it uses until you reload or restart. When your platform rotates pods, tasks, or VMs, the old IP is gone and the 502s are just nginx confidently dialing a corpse.

Why nginx upstream DNS cached after deploy bites you

In the default setup, a name inside a proxy_pass or an upstream block is resolved once, when the configuration is parsed. There is no background re-resolution and no TTL honor. That is deliberate — it keeps the request path free of DNS lookups — but it means a hostname in nginx behaves like a static IP with extra steps. The classic trigger is a short-lived container fleet behind a stable service name.

You can confirm the cause in two commands: dig +short <service-name> and compare it to the IP in your error log. Different address, same hostname, 502s only for the stale one.

Fix: force resolution per request with a variable

This is the change most teams want. The trick is that nginx only re-resolves when the upstream is a variable, which forces runtime lookup through a declared resolver.

1. Add a resolver pointing at your platform's DNS server: resolver 10.0.0.53 valid=5s ipv6=off; in the relevant server block. Keep valid short but not zero — match it to your deploy cadence.

2. Replace the literal upstream with a variable: set $backend "payments:8080"; then proxy_pass http://$backend;.

3. Because a variable proxy_pass without a URI part passes the original request URI through, verify path handling on a staging route before shipping.

4. Reload nginx and watch logs during a test scale-up. You should see fresh resolution, not a cached hit.

5. If you need a /path/ suffix in proxy_pass, use proxy_pass http://$backend$request_uri; or a rewrite.

The cost: with a variable you lose upstream-block features — load balancing across many servers, keepalive connection pools, passive health checks. That's often fine for a single service name behind a load balancer, and not fine when nginx is doing real balancing itself.

Alternatives worth knowing

If you run nginx Plus, or an open-source build recent enough to include the resolve parameter on upstream server directives [link needed], you can keep the upstream block and get periodic re-resolution without the variable workaround. Most distro packages are older than that, so check nginx -v first.

The other option is boring and effective: reload nginx as the final deploy step. nginx -s reload re-reads config and re-resolves, at the cost of a brief worker overlap. It's the right answer when you deploy rarely and can't restructure the config.

When NOT to do this

Don't add a resolver and start re-resolving per request when your upstream is genuinely static — bare-metal IPs, a fixed hardware LB, or a hostname whose DNS TTL is hours. You'd add a lookup to every request and gain nothing. Also skip it when nginx already runs behind an orchestrator service mesh that handles endpoint churn; you'd have two layers racing to resolve the same name.

Checklist

ad slot — paste the network tag here