Why your cron job silently stops and how to notice sooner
Why your cron job silently stops and how to notice sooner
At 2:15 a.m., the nightly invoice export exits before writing a file. The wrapper script returns 0, so cron records a successful run. Nobody notices until customers call at 9:00. The cost is not just one failed job; it is six hours of stale data, manual reconciliation, and a fire drill that could have been a one-line alert.
Silent cron failures usually look boring: a changed path, an expired token, a full disk, or a dependency that moved after an upgrade. The job may never start, or it may start and do nothing useful. Cron treats exit 0 as success and may email output only if mail is configured and working [link needed]. That is not detection; it is hoping someone reads the right mailbox. Cron also does not retry a missed run by itself [link needed], so “we’ll catch it next time” is often false.
Build cron job monitoring around outcomes
Schedule checks are weak. “Did cron fire?” misses the common case where it fired and produced garbage. Good cron job monitoring checks three things: the job ran, it produced the expected result, and it finished by a deadline.
For a nightly export, “expected result” might be a file newer than 26 hours with a nonzero row count. For a cleanup job, it might be a log line with a deletion count. For a sync, it might be a checksum match or an API response summary. Put those checks in the job wrapper or a separate verifier. Make the success signal conditional on passing them. Logs are for diagnosis after an alert, not detection.
A numbered procedure to catch silent failures
1. Wrap the command. Record start time, host, job name, and exit code. Write production output to a temporary path and move it only after validation.
2. Validate output. Check freshness, size, row count, or a business invariant. Exit nonzero if validation fails.
3. Send a success heartbeat only after validation. Use a cron job monitoring endpoint or a dead-man’s-switch service that expects a ping every run [link needed].
4. Alert on a missed heartbeat after the expected window. Alert on nonzero exits and validation failures too. Route alerts to the owning team or on-call, not a shared inbox.
5. Test the alert. Disable the job or force a validation failure, then confirm someone receives a useful message with job name, host, exit code, and runbook link.
Keep alert text short and actionable. “Invoice export missing; last success 2025-06-01 02:15; see runbook” beats “cron failed.”
Where cron job monitoring goes wrong
Do not page humans for every harmless job. A log rotation that can skip one night does not need the same urgency as payment capture. Use severity tiers: page for revenue or compliance jobs, ticket for routine maintenance, dashboard for best-effort tasks.
Also avoid heartbeat-only checks for jobs with legitimate variable schedules. If a job runs “every 2 hours during business hours,” a fixed 2-hour deadline will create false alarms [link needed]. Model the schedule or use freshness checks instead.
When NOT to do this
Skip custom cron job monitoring if your scheduler already provides missed-run alerts and output assertions, and those alerts reach the right people. Skip heartbeats for one-off scripts that no one depends on. Skip paging for jobs that can safely wait until business hours; use a daily digest instead. Do not add monitoring you will not maintain—stale alert rules are worse than none.
Checklist
- [ ] Success is sent only after output validation
- [ ] Missed heartbeat alerts after a defined deadline
- [ ] Alerts include job name, host, exit code, and runbook
- [ ] Owner or on-call receives alerts
- [ ] Failure path tested by disabling the job
- [ ] Severity matches business impact