Systemd unit starts before /mnt/data is mounted after reboot, so app exits with ENOENT
Systemd unit starts before /mnt/data is mounted after reboot, so app exits with ENOENT
After a reboot, app.service exits with ENOENT on /mnt/data/config.json. The service starts, logs one line, and stops; seconds later mnt-data.mount becomes active. The cost is a broken app after every reboot and noisy alerts while the host is otherwise healthy. This is usually a systemd After mount ordering gap: the unit has no ordering or requirement against the mount unit, so systemd is free to start it independently.
Confirm the mount is actually late
First prove the order, not just the error.
- Run
systemctl status app.service --no-pagerand note its start time and exit code. - Run
systemctl status mnt-data.mount --no-pagerandfindmnt /mnt/data. If the mount is inactive or failed when the service starts, you have an ordering issue [link needed]. - Compare journals for this boot:
journalctl -b -u app.service -u mnt-data.mount --no-pager. - If
/mnt/datais in/etc/fstab, systemd generates a mount unit throughsystemd-fstab-generator; the unit name derives from the path [link needed]. Confirm it withsystemctl list-units --type=mount --all. - Check
systemctl cat app.servicefor existingAfter=orRequires=lines. Drop-ins merge, so a partial fix may already exist.
Do not add ExecStartPre=/bin/sleep 10 yet. It hides order without fixing it.
Fix the dependency, not the delay
Use a numbered procedure. Edit a drop-in, not the distribution unit.
1. Create an override: sudo systemctl edit app.service.
2. In [Unit], add RequiresMountsFor=/mnt/data. This adds systemd After mount ordering plus a requirement for the mount units needed to access that path [link needed].
3. If explicit names are required, use both After=mnt-data.mount and Requires=mnt-data.mount. After= alone orders starts but does not fail the service when the mount fails [link needed].
4. Reload systemd: sudo systemctl daemon-reload.
5. Verify with systemctl show -p After -p Requires -p RequiresMountsFor app.service and systemd-analyze critical-chain app.service. The mount should appear before the service [link needed]. Also run systemd-analyze verify app.service to catch misspelled unit names.
6. Reboot and compare journal timestamps. The ENOENT should disappear when the mount succeeds.
If the app runs as a user unit, put equivalent dependencies in that unit and confirm the mount is visible in the user manager’s namespace [link needed].
Use systemd After mount ordering deliberately
After= controls start order only; it does not pull the mount into the transaction. RequiresMountsFor= is usually the practical choice because it derives dependencies from a path instead of hard-coding a unit name. Use BindsTo= only when the service must stop if the mount disappears [link needed]. Keep mount options in /etc/fstab or a .mount unit, and keep app-specific ordering in the service drop-in. That separation makes systemctl list-dependencies and systemd-analyze useful during review.
When NOT to do this
Do not hard-require the mount if /mnt/data is optional and the app can serve degraded mode. Use After=mnt-data.mount without Requires=, or let the app retry, so a failed mount does not take down the service [link needed]. Skip RequiresMountsFor= when the mount depends on the service starting first; that creates an ordering cycle. Also skip it when the mount point is created by the app itself or is an automount triggered on first access; forcing it active at service start can defeat the automount design [link needed]. In those cases, systemd After mount ordering is not enough by itself, but it may still be the wrong layer to enforce.
Checklist
- [ ] Confirm mount and service timestamps from this boot.
- [ ] Identify the exact mount unit with
systemctl list-units --type=mount --all. - [ ] Add
RequiresMountsFor=/mnt/dataor explicitAfter=plusRequires=. - [ ] Run
systemctl daemon-reload. - [ ] Check
systemctl showandsystemd-analyze critical-chain. - [ ] Reboot and verify no
ENOENT.