When log rotation deletes the file your app still holds open

2026-11-09 · 3 min read

When log rotation deletes the file your app still holds open

At 03:12 the disk alarm fires: /var at 100%. You check the log directory, and the file you expect — app.log, 14 GB this morning — is gone. Rotation ran an hour ago and "cleaned up." But df still shows the space consumed, and du on the directory now disagrees with it. Nothing you delete frees anything, because the process writing that log opened it days ago and never let go.

That's the failure. The cleanup succeeded; the disk stayed full. If the volume is small and the app writes fast, you get crashes or a read-only filesystem before the follow-up rotation.

Why deleting the file doesn't return the space

On a normal Linux filesystem, a file's data lives on while anything still holds a reference to its inode. rm removes the directory entry and decrements the link count; the blocks are only released when the last open file descriptor closes.

So a rename-and-recreate rotation is safe only if the writer reopens the path afterwards. Most daemons do — after SIGHUP or on their own. Plenty of application runtimes don't: a shell redirect, a Python process holding open('app.log', 'a'), a JVM logger configured with a fixed path. Those keep appending to the inode that now has no name. Disk usage stays flat at the old high-water mark, and the fresh app.log stays empty. You can confirm this with lsof +L1, which lists unlinked files still held open [link needed].

What logrotate copytruncate actually does

logrotate copytruncate avoids the reopen problem entirely. Instead of renaming, logrotate copies the file's contents to the rotated name and then truncates the original to zero bytes in place. The inode never changes, so the application's file descriptor stays valid and its writes land in the file the next rotation will read. No signal, no restart, no cooperation from the app.

Two properties follow, and both matter:

There's a subtler issue: truncate sets the size to zero but doesn't move the writer's file offset. A process that opened the file without O_APPEND and keeps its position will write at the old offset, leaving a sparse hole of NUL bytes at the front. Log parsers handle it, but tail shows blank lines [link needed]. Appenders are unaffected.

A procedure to switch a stuck log safely

1. Identify the exact path and the process holding it: lsof /path/to/app.log.

2. Check free space; if the volume is under about twice the log size, shrink the log first or move it to another volume.

3. In the logrotate stanza, add copytruncate and remove create (it's redundant). Keep rotate and maxsize as they are.

4. Test with logrotate -d /etc/logrotate.d/app to see the plan, then logrotate -f to force one cycle.

5. Verify: lsof shows the same inode still open, ls -l shows the file near zero bytes, df drops, and new lines still appear.

6. Watch for NUL padding at the start of the new file; if you see it, the writer isn't opening with O_APPEND [link needed].

When NOT to use logrotate copytruncate

Skip it when logs are very large and traffic is heavy — the copy cost and the loss window both scale, and you may prefer create plus a signal to the app. Skip it when the software has a supported reopen mechanism (SIGHUP, a management command, a built-in rotating appender); those are cleaner. And skip it when the volume genuinely can't hold a second copy, since copytruncate will fail mid-cycle [link needed]. Also note it only works for whole files, not for wildcard-matched chained logs.

Checklist

ad slot — paste the network tag here