Kubernetes pod stuck in CrashLoopBackOff because the liveness probe fires before the app finishes starting
Kubernetes pod stuck in CrashLoopBackOff because the liveness probe fires before the app finishes starting
The rollout is at 60%. Two of five new pods are in CrashLoopBackOff, restart counts climbing, and the deploy job will time out in four minutes. The app is fine — the JVM just needs 45 seconds to load a config bundle, and your liveness probe starts hitting /healthz after 10. The kubelet reads connection-refused as "dead," kills the container, and starts over.
Why a slow start looks exactly like a crash
A liveness probe failure does one thing: it triggers a container restart, regardless of restartPolicy: Always. If the probe's failureThreshold is exhausted before the process can listen, the kubelet kills a container that was still booting. Each restart re-enters the same window, so the pod never escapes. After a few failures the pod sits in CrashLoopBackOff with exponential backoff (10s, 20s, 40s, capped at 5 minutes) [link needed], which makes the cycle look even more like a genuine crash.
The diagnostic tell is in the events, not in the app logs. kubectl describe pod shows a Liveness probe failed: connection refused immediately ahead of the Killing event, and the app's own log never reaches its "started" line.
Repair it with a startup probe first
Kubernetes added startupProbe for precisely this case: while it runs, liveness and readiness checks are disabled. Give it a generous failureThreshold, so the allowed startup budget equals failureThreshold × periodSeconds (60 × 5s = 5 minutes, for example) [link needed]. Then keep the liveness probe strict, so it still catches real hangs later.
Adjusting kubernetes liveness probe initialDelaySeconds alone also works, but it slows every restart's detection window and drifts out of date as startup times change. Prefer the startup probe; treat initialDelaySeconds as a secondary lever.
1. Confirm the cause: kubectl describe pod <pod> and check that a liveness failure precedes each kill.
2. Time a real cold start locally or from a known-good pod's logs.
3. Add a startupProbe with an HTTP or exec handler and a budget roughly 2× your measured startup time.
4. Keep the existing liveness probe, but verify its periodSeconds and failureThreshold give at least 15–20 seconds of grace for transient stalls.
5. Set kubernetes liveness probe initialDelaySeconds to a small value (0–10) for the liveness probe, since the startup probe now covers boot.
6. Roll out to a canary; watch kubectl get events -w for probe failures during startup.
7. Only widen timeouts after confirming the app genuinely needs them, not because the first attempt failed.
Tuning kubernetes liveness probe initialDelaySeconds without masking hangs
The value is a floor, not a fix. Setting kubernetes liveness probe initialDelaySeconds: 120 on every container buys silence, but a truly wedged process now takes two minutes longer to restart, and every pod holds resources through the delay. Base the number on a p95 startup measurement, add margin for cold caches, and re-measure after dependency changes.
When NOT to do this
- The app never becomes healthy. If startup takes longer than any reasonable budget, a longer
initialDelaySecondsjust hides a startup-time regression; profile the boot path instead. - The container exits on its own. A probe delay does nothing for an app that crashes during init — check exit codes first.
- You're on a pre-1.16 cluster. No
startupProbe; you're limited toinitialDelaySecondsand a loosefailureThreshold[link needed]. - The probe checks a dependency. If
/healthzpings the database, a slow dependency will restart otherwise-healthy pods; split liveness from readiness.
Checklist
- [ ] Liveness failure confirmed in
kubectl describe podevents before each kill - [ ] Cold-start time measured, not guessed
- [ ]
startupProbeadded with a 2× budget - [ ]
kubernetes liveness probe initialDelaySecondsreduced to near zero for liveness - [ ] Liveness
failureThreshold×periodSeconds≥ 15s - [ ] Readiness probe restricted to traffic, not dependencies
- [ ] Canary rolled out and watched before full deploy