Rsync backup fails with permission denied after NFS export enables root_squash
Rsync backup fails with permission denied after NFS export enables root_squash
Your backup job has run clean for months. Then a storage change lands, and the next run dies on the first file with rsync: mkstemp "/mnt/backup/var/www/.index.html.XXXXXX" failed: Permission denied (13). The job exits non-zero, the snapshot is half-written, and the monitor only tells you it failed — not why. If the change was an NFS export option, the cause is almost certainly root_squash.
What root_squash does to an rsync run
root_squash is an export option in /etc/exports. It tells the NFS server to map the client's uid 0 to the anonymous uid (typically 65534, nobody) [link needed]. On the server side, every file operation that the client sends as root is now performed as nobody.
That matters because rsync run as root relies on being root. It creates temporary files (mkstemp) in the destination directory, and it tries to chown files back to their original owners when preserving attributes. Under root_squash, the server evaluates those operations as nobody, so a destination directory owned by root:root with mode 0755 is not writable, and every chown to another uid fails. The classic symptom of rsync permission denied root_squash is that the failure appears immediately on the *first* file and repeats for every file after it, while reads succeed fine.
Confirm the cause before changing anything
Run showmount -e <server> or read the export entry on the server and look for root_squash or all_squash. Then check the destination: ls -ld /mnt/backup/var/www. If it is owned by root with a restrictive mode and you are connecting as root, you have reproduced it. Also confirm you are actually root on the client with id -u; an rsync running as a normal user fails the same way for a different reason.
Fix a failing rsync permission denied root_squash job
Work through these in order and stop when the job passes.
1. Prove the diagnosis. Run one file manually with rsync -av --dry-run disabled and a single source file to the destination. Reproduce the error deliberately so you know the fix works.
2. Make the destination writable by the mapped user. Either chown nobody:nogroup the destination tree, or open the directory mode. Ownership of the files inside matters less than the directory bit, because mkstemp needs write on the directory.
3. Handle ownership separately. With -a, rsync still attempts chown, which nobody cannot do to arbitrary uids. Either add --no-owner --no-group to the job, or accept the failures and check whether rsync exits non-zero anyway. Use --numeric-ids if you keep ownership and both sides share the same uid map.
4. Or adjust the export. If the client is trusted, change the export to no_root_squash for that host, then run exportfs -ra on the server [link needed]. This restores previous behaviour immediately.
5. Re-run the real job and confirm a zero exit code plus a non-empty destination, before you trust the next scheduled run.
When NOT to do this
Do not switch to no_root_squash just because it is the fastest fix. If the NFS export is shared with untrusted clients, a root user on any of them can chown and overwrite anything on the export, so you widen the blast radius for each one. Do not use --no-owner either if downstream tooling depends on the restored uid map — you will get files that look correct but are owned by nobody, and a restore that silently changes ownership is worse than a loud failure.
Checklist
- [ ]
root_squashconfirmed in the export or viashowmount -e - [ ] Destination directory writable by the anonymous uid
- [ ] Ownership handled with
--no-owner --no-groupor--numeric-ids - [ ] If
no_root_squashwas chosen, every client on that export reviewed - [ ] Full job re-run to a zero exit code
- [ ] Monitoring alerts on non-zero rsync exits, not just missing snapshots