A health check that returns 200 while Postgres is unreachable
A health check that returns 200 while Postgres is unreachable
Your load balancer keeps sending traffic to an app instance whose /health returns 200 because the HTTP server is alive. Postgres is down. Every request that needs a query returns 500, but the orchestrator sees “healthy,” so it does not remove the instance or restart it. The outage is visible to users and invisible to the health system. A health check database connectivity probe fixes that gap by making the endpoint depend on a real database round trip.
Why a process-only health check lies
Most frameworks’ default health route proves only that the event loop can answer. It does not acquire a database connection. If Postgres is unreachable, DNS fails, credentials rotate, or the pool is exhausted, the endpoint still returns 200. Kubernetes readiness probes are meant to decide whether a pod should receive traffic [link needed]; a process-only readiness check keeps broken pods in rotation. Liveness is different: it decides whether to restart the process [link needed].
Make the health check database connectivity probe fail correctly
Keep liveness cheap and keep readiness honest.
1. Split endpoints. /live returns 200 if the process can serve HTTP. /ready includes the database check. Point liveness at /live, readiness at /ready [link needed].
2. In /ready, acquire a connection with a short context timeout. Run SELECT 1. Close the connection. Do not reuse a cached boolean.
3. Return 503 on timeout, auth failure, DNS failure, or pool exhaustion. Return 200 only after the query succeeds. Keep the body small: {"db":"ok"} or {"db":"unavailable"}. Do not leak credentials.
4. Set the probe timeout below the endpoint’s DB timeout budget, and set failureThreshold so one transient blip does not remove every instance [link needed].
5. Test it. Stop Postgres or revoke connect permission in staging. Confirm /ready returns 503, the instance leaves the load balancer, and traffic recovers after restore.
6. Alert on readiness failures, not just process restarts. Log the database error class with the instance ID.
Example handler shape:
func ready(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 750*time.Millisecond)
defer cancel()
if err := db.PingContext(ctx); err != nil {
w.WriteHeader(http.StatusServiceUnavailable)
io.WriteString(w, `{"db":"unavailable"}`)
return
}
w.WriteHeader(http.StatusOK)
io.WriteString(w, `{"db":"ok"}`)
}
PingContext acquires a connection and verifies it [link needed]. A SELECT 1 works too and is easier to see in query logs.
Common false passes
A health check can still lie. Checking only that port 5432 accepts TCP does not prove auth or query execution. Checking pool statistics can show idle connections while the database rejects new ones. Caching the last successful result hides an outage until the cache expires. A timeout longer than the probe timeout lets the orchestrator mark the probe failed before your handler returns. And returning 200 when connection acquisition fails is the original bug in a new place.
When NOT to do this
A health check database connectivity probe is not always right for liveness. If Postgres is down, including it in liveness restarts every app instance even though restarting will not fix the database [link needed]. For readiness, if the app can serve cached or degraded traffic, a hard DB check can remove every instance and turn a partial degradation into a full outage. In that case, use a separate degraded readiness signal or check only the dependency needed for the route. Do not run heavy queries or migrations in the probe.
Checklist
- [ ]
/livechecks only the process. - [ ]
/readyacquires a connection and runsSELECT 1. - [ ] DB timeout is shorter than the probe timeout.
- [ ] Failure returns 503; success returns 200.
- [ ] Failure removes the instance from traffic.
- [ ] Recovery returns the instance to traffic.
- [ ] Alerts fire on readiness failures.