Kubernetes pod CrashLoopBackOff after ConfigMap update because app reads config only at startup

2026-11-13 · 3 min read

Kubernetes pod CrashLoopBackOff after ConfigMap update because app reads config only at startup

At 02:14 a one-line edit to a ConfigMap lands: max_connections: 500 becomes max_connections: 5OO (letter O). The next pod that starts parses the file, throws on the invalid integer, and exits 1. Kubernetes restarts it, it fails again, and within a minute kubectl get pods shows CrashLoopBackOff with a backoff already climbing past two minutes. Nothing in the ConfigMap is "broken" from Kubernetes' point of view — the object was accepted — so no rollout, alert, or admission check fires until pods start dying. That gap between "config applied" and "pods that must restart to read it" is where most of these incidents live.

The mechanics of the loop

Three facts combine badly. First, a container that exits non-zero is restarted by the kubelet with exponential backoff, capped at five minutes [link needed]. Second, ConfigMap data mounted as a volume is eventually refreshed, but not instantly, and files mounted with subPath are never refreshed at all [link needed]. Third, ConfigMap keys injected as env vars only reach a container at start time [link needed].

An app that reads and validates config once during boot therefore has exactly two failure shapes: it crashes and restarts against the same bad data forever, or it keeps running on stale values until someone deletes the pod. An app that re-reads on SIGHUP or on a file watcher fails differently — it may log errors without ever entering CrashLoopBackOff, which is quieter and often worse.

Diagnosing a Kubernetes CrashLoopBackOff ConfigMap change

Work the evidence in order; don't restart anything until step 5.

1. Confirm the state and the reason: kubectl get pod <name> -o wide then kubectl describe pod <name>. You want Last State: Terminated, Reason: CrashLoopBackOff, and the exit code.

2. Read the dead container's output, not the current one: kubectl logs <name> --previous. The current container may not have logged anything yet.

3. Diff the mounted config inside a fresh container: kubectl exec -it <name> -- cat /etc/app/config.yaml (or start a debug pod mounting the same ConfigMap). If the content differs from what you applied, suspect subPath or a kubelet resync window [link needed].

4. Check whether the app validates at boot: kubectl exec -it <name> -- sh -c 'grep -n "read.*config" /app/*.py' or read the entrypoint. If config loading is in the startup path, a bad value is fatal by design.

5. Fix forward: correct the ConfigMap, then kubectl rollout restart deployment/<name>. Flipping back to the old value also works and is faster under pressure.

6. Watch the exit code change from failure to 0 and confirm the backoff resets before you close the incident — a pod can recover on its own and still hide the root cause.

Reducing the blast radius

Make the failure visible earlier than the reboot. Options, in rough order of payoff:

When NOT to do this

Don't wire a checksum-triggered restart into a ConfigMap that changes on a schedule for reasons unrelated to code — you'll get rolling restarts of a whole Deployment every few minutes for a value nobody reads. Also skip the restart-on-change pattern for anything mounted with subPath, since the kubelet won't refresh those files and a restart won't help unless the pod is recreated [link needed]. And if the app already handles SIGHUP correctly, a full rollout is heavier than sending the signal.

Checklist

ad slot — paste the network tag here