Logrotate deletes the open access.log and the app keeps writing to the unlinked inode until the disk fills

2026-11-13 · 3 min read

Logrotate deletes the open access.log and the app keeps writing to the unlinked inode until the disk fills

At 03:12, the disk alarm fires. df -h / is at 100%, but du -sh /var/log/* shows only a few megabytes, and access.log itself is nearly empty. sudo lsof +L1 | grep access.log shows the missing space: the app still holds a deleted access.log open and is writing to that unlinked inode. Restarting the app releases the space, but the next rotation repeats the failure.

Why a logrotate deleted inode disk full happens

A rotation that renames access.log to access.log.1 and creates a new access.log does not update file descriptors already held by the app. The app keeps writing to the old inode through its open fd. When later rotations exceed the rotate count, logrotate deletes access.log.1. The name disappears, but the inode remains allocated while the fd is open. du on the directory may not count it, while df does.

A logrotate deleted inode disk full event is checkable in two places: lsof +L1 lists files with link count zero, and Linux exposes deleted targets under /proc/<pid>/fd as symlinks ending in (deleted).

Find and recover the deleted inode

1. Confirm the open deleted file. Run as root:

```bash

sudo lsof +L1 | grep -E 'access\.log|deleted'

sudo ls -l /proc/<pid>/fd | grep deleted

```

2. Identify the owning service from the PID:

```bash

ps -fp <pid>

systemctl status <service>

```

3. Make the app reopen its log. Prefer the app’s supported reload. For nginx, sudo kill -USR1 $(cat /run/nginx.pid) reopens logs; for Apache, use a graceful restart. For a custom app, use its documented reload or restart.

4. Verify the fd is gone:

```bash

sudo lsof +L1 | grep access.log

df -h /

```

Space should drop once the last fd closes.

5. Fix the logrotate rule, then test it:

```bash

sudo logrotate -d /etc/logrotate.d/app

```

Fix the config so it does not recur

If the app can reopen logs, keep rename+create and add a postrotate reload. If several logs share the rule, use sharedscripts so the app reloads once:


/var/log/app/access.log {
  daily
  rotate 14
  missingok
  notifempty
  compress
  delaycompress
  create 0640 www-data adm
  sharedscripts
  postrotate
    /usr/bin/systemctl reload app >/dev/null 2>&1 || true
  endscript
}

If the app cannot reopen logs, use copytruncate. It copies the file, then truncates the original, so the app’s fd stays valid and points at the same inode:


/var/log/app/access.log {
  daily
  rotate 14
  missingok
  notifempty
  compress
  delaycompress
  copytruncate
}

copytruncate has a race: lines written during the copy window can be lost or duplicated [link needed]. Use it when the alternative is a logrotate deleted inode disk full outage.

When NOT to do this

Do not use copytruncate for very large logs or high write rates where the copy takes long and losing a few lines matters; reopen the log instead [link needed]. Do not manually rm a rotated file while the app still holds it open unless you accept that space stays allocated. Do not add a reload signal that the app does not handle; test with logrotate -d first.

Checklist