Why your cron job silently stops and how to notice sooner: the failure cases to plan for
Why your cron job silently stops and how to notice sooner: the failure cases to plan for
At 02:10, the nightly export writes a zero-byte CSV and exits 0. The process ran, so nothing pages. By 09:00, support has tickets, the finance team has stale numbers, and someone is manually rerunning a job that “worked fine yesterday.” Without cron job monitoring, the first signal is usually a customer or a colleague.
Silent stops are rarely dramatic. They look like success: an exit code of 0, a log line saying “done,” or a scheduler entry that still exists.
The failures that look like success
Plan for these before they happen:
- The script exits 0 after writing an empty file or partial output.
- A stale lock file makes every new run skip.
- A credential or token expires, and the script catches the error but still exits cleanly.
- The disk fills; the job logs the failure but returns 0.
- The cron daemon is disabled after a reboot, or the host is decommissioned.
- A path, environment variable, or timezone differs between interactive runs and cron.
- The job runs longer than its interval, so runs overlap or queue up.
Cron tells you a command was launched, not that it finished usefully. Implementations also differ in whether they mail command output at all [link needed].
Build cron job monitoring around outcomes
A useful cron job monitoring setup checks the result, not just the schedule. Follow this procedure:
1. Define success as something observable. Examples: a file newer than the last scheduled run, a row count above zero, a database row updated, or a successful API response.
2. Emit a heartbeat on success. Write a timestamp, job name, duration, and result to a local file or an external endpoint.
3. Check freshness from outside the host. Alert when the last success is older than the schedule plus a grace period. If the host is down, a local-only check cannot tell you.
4. Capture stderr and non-zero exits. Treat them as one signal, not the only signal.
5. Test the failure path. Disable the job or force an empty output once a month and confirm the alert arrives.
Keep the heartbeat small and append-only. If the job writes to a database, update a last_success_at row in the same transaction as the work where possible.
Make alerts specific enough to act on
An alert should include the job name, host, last successful run, expected schedule, first error line, and a runbook link. Route it to the owning team. Deduplicate repeated failures so a flapping job does not train people to ignore the channel.
Use two thresholds: a warning after the first missed window and a page after two or three. Send a separate “recovered” notice. Check alert delivery independently; an alerting path that depends on the same host as the job will fail with it.
When NOT to do this
Do not build full cron job monitoring for a one-off personal script whose failure is obvious and cheap. A log line is enough. Skip paging for best-effort jobs that can safely wait for a daily digest. If a job runs more often than it can finish, fix the lock or scheduling first; more alerts will not solve overlap. And do not monitor only from the job’s own host if that host disappearing is one of your failure cases.
Checklist before you trust the schedule
- Success has a concrete, observable definition.
- A success heartbeat is emitted.
- Freshness is checked from outside the host.
- Stderr and non-zero exits are captured.
- Alerts name an owner and a runbook.
- The failure path has been tested recently.
- Warning and page thresholds are separate.
- Someone reviews stale alerts and removes noise.