Detecting a disk filling up before writes start failing
Detecting a disk filling up before writes start failing
A cron job uploads backups every night at 02:00. At 01:47 the database starts writing to a full volume, and every INSERT returns ENOSPC. By the time anyone opens a dashboard, the transaction log is corrupt, the application has been restarting in a loop for four hours, and you are restoring from yesterday's dump. The disk had been creeping toward full for nine days. Nobody had a threshold that would have told them.
Why "disk full" is a lagging indicator
Most monitoring systems are built around state, not trajectory. A disk at 61% is "healthy" until it is at 96%, and only then does the alert fire — long after the last useful moment to act. The write path fails at exactly 100%, so an alert at 95% gives you a window measured in minutes on a busy host and days on a quiet one. Neither is a plan.
The fix is to alert on time-to-full, not on percentage. Fit a line to the free-space samples over the last several hours and ask how long until the line hits zero. A volume dropping 2 GB/hour with 40 GB free is nine hours from failure and deserves attention right now, even though it is sitting at a comfortable-looking 80%.
A procedure you can follow
1. Check that your agent collects free bytes per mount point at a fixed interval. One- to five-minute resolution is enough; more frequent sampling adds noise.
2. Store the samples with timestamps in a time-series store that retains at least 30 days, so short spikes and long ramps are both visible.
3. For each mount with a history of at least six hours, compute the slope of free space using a linear fit over a trailing window. Trim the window so single large writes do not dominate.
4. Derive a projected exhaustion timestamp from the slope and current free bytes. Ignore negative slopes smaller than a noise floor you calibrate per host.
5. Alert when the projection is closer than your remediation time — the time it takes a human to log in, identify the consumer, and delete or extend something.
6. Keep a static floor alert as a fallback for flat-but-low volumes, at a percentage high enough to leave room for logs to grow.
7. Route the alert to whoever owns the host, and include the growth rate, top consumers, and largest recent files in the payload.
8. Silence the projection alert for volumes that are deliberately shrinking, or you will page every night during a rotation.
Getting disk space alerting right for the whole fleet
A projection model will page you on a host that is only full because a log directory was not rotated, and it will miss the host that filled 100 GB in ninety seconds. Run both signals. Good disk space alerting pairs the slope-based projection with a rate-of-change guard that fires when consumption over any five-minute window exceeds a threshold you set from the host's normal write pattern.
Also alert on inode usage on filesystems that store many small files. A disk can read as 40% used and still refuse to create a file when it runs out of inodes [link needed].
Per-host thresholds beat global ones. A build server with an ephemeral scratch volume and a database primary should not share a percentage. Override at the host level and keep the default moderately conservative.
When NOT to do this
Skip projection alerting on ephemeral, single-purpose hosts that are already replaced on failure — short-lived CI runners, autoscaled batch nodes, containers with a capped writable layer. The projection will not converge before the instance is gone, and the pages are noise. For those, alert only when the instance is still alive and the volume crosses an absolute floor.
Skip it too on volumes that are designed to stay near full, such as thin-provisioned pools where 90% is the steady state. There, watch the underlying pool's free capacity rather than the guest's.
Checklist
- Free bytes sampled per mount at a fixed interval
- Slope fitted over a trailing window with a noise floor
- Projected exhaustion alert, tuned to remediation time
- Static floor alert as a fallback
- Inode alert on filesystems with many small files
- Rate-of-change guard for fast fills
- Exclusions for ephemeral and deliberately-full volumes
- Alert payload includes growth rate and top consumers