harden TrueNAS CIFS mounts so immich self-heals

TrueNAS was power-cycled, the CIFS mounts failed, and systemd never retried
-- mount units are not restarted on failure. SMB came back, nothing
remounted, and immich-server served an empty library for days while its
database still listed 14,181 assets pointing at /mnt/media/originals.

fstab carried no _netdev, no nofail and no automount, so there was no path
back without a human. Now:

  x-systemd.automount  any access re-attempts the mount; failure stops
                       being terminal
  _netdev / nofail     ordered after network-online, dead NAS cannot block boot
  soft                 I/O errors instead of blocking forever, so the
                       container can be restarted rather than wedging in
                       uninterruptible sleep
  idle-timeout         unmount when unused, clearing stale handles
  resilienthandles     SMB3 rides out brief blips

ansible.posix.mount mounts directly and never starts the generated
.automount unit, leaving the on-access trigger inactive -- enable it
explicitly, or the headline fix silently does nothing.

The containers are systemd USER units while the mounts are SYSTEM units, so
RequiresMountsFor= is unavailable. cifs-watchdog bridges the scopes: checks
health, recovers, and restarts ONLY immich-server (the sole consumer of both
paths; postgres/redis/ML use local volumes).

Two bugs the umount test caught, both worth knowing:
- `ls` cannot test mountedness. An unmounted mount point is an ordinary
  empty directory, so ls succeeds and recovery was skipped entirely.
- A drop repaired within a single run leaves prev=healthy, so keying the
  restart solely on the stored state skipped it while the container still
  held its stale view.

Also moves the SMB password out of /etc/fstab, which is 0644 and was
readable by every local user, into a 0600 credentials file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
2026-08-02 14:49:22 -04:00
parent 0939204061
commit 773a2bbc9c
8 changed files with 242 additions and 2 deletions
+11
View File
@@ -213,3 +213,14 @@ podman_prune_users:
# Keep 30 days of unused images so a rollback needs no rebuild or re-pull.
podman_prune_until: 720h
podman_prune_oncalendar: "Sun *-*-* 02:00:00"
# Hardened CIFS options for the TrueNAS shares (see containers/home/photos.yml).
# x-systemd.automount is what makes a failed mount recoverable without a human.
cifs_mount_opts: >-
credentials=/etc/cifs/photos.creds,uid={{ podman_subuid.stdout }},gid={{ podman_subuid.stdout }},_netdev,nofail,soft,x-systemd.automount,x-systemd.mount-timeout=30,x-systemd.idle-timeout=600,vers=3.1.1,resilienthandles,echo_interval=10
# How often the watchdog checks mount health and heals immich.
cifs_watchdog_oncalendar: "*:0/5"
cifs_watchdog_mounts:
- "{{ photos_path }}/storage"
- "{{ photos_path }}/immich"