Journald fills the disk because SystemMaxUse is unset and defaults to 10% of the filesystem

2026-11-14 · 3 min read

Journald fills the disk because SystemMaxUse is unset and defaults to 10% of the filesystem

A 200 GB VPS crosses 95% disk usage on a Tuesday night. Nothing in /var/www grew. Nothing in /var/log grew either — not the rsyslog files, at least. The culprit is /var/log/journal, and it got that big because nobody ever set a limit on it: SystemMaxUse was left commented out in /etc/systemd/journald.conf, so journald did what its defaults told it to do and allowed the journal to consume a large slice of the filesystem.

That default is by design, not a bug. journald.conf(5) documents SystemMaxUse= as defaulting to 10% of the size of the respective file system, capped at 4 GiB [link needed]. On a 200 GB volume, 10% is 20 GB of binary journal — before the cap even matters. On a 1 TB volume, the cap saves you from 100 GB. Either way, a full disk is a full disk: Postgres can't write WAL, logrotate starts failing, and the package manager refuses to run. The fix is one config line, and it takes less time than reading this paragraph.

Confirm the journal really is the problem

Before changing anything, measure. journalctl --disk-usage prints the current on-disk size of the journal, and du -sh /var/log/journal should agree. Then check the effective setting with systemd-analyze cat-config systemd/journald.conf — this shows the merged config including drop-ins, and it will show SystemMaxUse absent or commented if the default is in play. If the journal is tens of gigabytes while /var/log from rsyslog is measured in megabytes, you have found your leak.

Run journalctl --verify too. A corrupted journal can slow everything down, and you want to know before you start rotating files around.

Set a hard cap with journald SystemMaxUse

The durable fix is a drop-in file, not an edit to the package-owned main config — vendor updates overwrite /etc/systemd/journald.conf less often than you'd think, but drop-ins survive them cleanly.

1. Create the drop-in directory: mkdir -p /etc/systemd/journald.conf.d.

2. Write /etc/systemd/journald.conf.d/99-size-limit.conf containing a [Journal] header and a line like SystemMaxUse=2G. Size it against what you actually need for debugging — 1–4 GB covers weeks of logs on most single-purpose servers.

3. Add SystemMaxFileSize=128M so no single journal file grows unreasonably large; the default derives from SystemMaxUse [link needed].

4. If the journal lives on a separate small volume, set SystemMaxUse under that volume's size, or use RuntimeMaxUse= for /run limits.

5. Reload with systemctl restart systemd-journald. Rotation does not depend on the restart alone — the limit is enforced on rotation — so force it: systemctl kill -s USR2 systemd-journald.

6. Trim the backlog immediately: journalctl --vacuum-size=2G.

7. Verify with journalctl --disk-usage and systemd-analyze cat-config systemd/journald.conf.

On hosts where you only want a floor-level safety net, journalctl --vacuum-time=2weeks in a weekly timer is the crude version of the same thing. Nothing about a journald SystemMaxUse disk full incident requires a reboot, a reinstall, or a monitoring vendor.

Watch for the follow-on failures

A journal that ate the disk usually took other things with it. Check whether rsyslog or syslog-ng is also configured and whether anything else writes to /var/log. Confirm the package database still has room by running apt-get check or dnf check. If /var was the volume that filled, verify no files were truncated mid-write.

When NOT to do this

Don't hard-cap the journal aggressively on a host you debug retroactively — an application server where the last three weeks of logs are the only forensic record. A 500 MB cap on a chatty service can mean a few hours of history, and once it's vacuumed, it's gone. If your organization requires log retention for compliance, the journal limit needs to account for the retention window, not just for free space, and you probably want persistent export to rsyslog as well. Also skip the cap on ephemeral containers where the journal is already volatile — you're tuning a file that disappears at stop.

Checklist