harden TrueNAS CIFS mounts so immich self-heals

TrueNAS was power-cycled, the CIFS mounts failed, and systemd never retried -- mount units are not restarted on failure. SMB came back, nothing remounted, and immich-server served an empty library for days while its database still listed 14,181 assets pointing at /mnt/media/originals. fstab carried no _netdev, no nofail and no automount, so there was no path back without a human. Now: x-systemd.automount any access re-attempts the mount; failure stops being terminal _netdev / nofail ordered after network-online, dead NAS cannot block boot soft I/O errors instead of blocking forever, so the container can be restarted rather than wedging in uninterruptible sleep idle-timeout unmount when unused, clearing stale handles resilienthandles SMB3 rides out brief blips ansible.posix.mount mounts directly and never starts the generated .automount unit, leaving the on-access trigger inactive -- enable it explicitly, or the headline fix silently does nothing. The containers are systemd USER units while the mounts are SYSTEM units, so RequiresMountsFor= is unavailable. cifs-watchdog bridges the scopes: checks health, recovers, and restarts ONLY immich-server (the sole consumer of both paths; postgres/redis/ML use local volumes). Two bugs the umount test caught, both worth knowing: - `ls` cannot test mountedness. An unmounted mount point is an ordinary empty directory, so ls succeeds and recovery was skipped entirely. - A drop repaired within a single run leaves prev=healthy, so keying the restart solely on the stored state skipped it while the container still held its stale view. Also moves the SMB password out of /etc/fstab, which is 0644 and was readable by every local user, into a 0600 credentials file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs(podman): add role README with backup, restore and LibreSign runbook
2026-08-02 14:49:22 -04:00 · 2026-08-01 15:55:04 -04:00 · 2026-08-01 15:54:52 -04:00 · 2026-08-01 15:54:27 -04:00
23 changed files with 1786 additions and 44 deletions
@@ -0,0 +1,143 @@
+# podman role
+
+Container orchestration for the home server. Containers are defined under
+`tasks/containers/{base,home,skudak,debyltech}/` and wired from
+`tasks/main.yml`, which is where every image tag is pinned.
+
+## Backups
+
+Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one
+backup engine: `tasks/containers/cloud-backup.yml` plus
+`templates/nextcloud/cloud-backup.sh.j2`. Each instance includes it with its own
+vars, producing `/usr/local/bin/<name>-backup.sh` and a systemd timer.
+
+Stages, in order — the ordering is deliberate, see the comments in the template:
+
+1. **Database dump** inside the container (mariadb / mysql / postgres branches),
+   gzipped to `/var/backups/nextcloud/<name>/db/`. Promoted over yesterday's dump
+   only after passing `gzip -t` **and** a completion-trailer grep.
+2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used.
+3. **rsync** of the data tree, config, and db dumps to TrueNAS.
+
+Failures raise `status=failed` on the `nextcloud-backup` syslog tag, which an
+external Graylog rule matches to send mail. Do not rename that tag: it is shared
+by every instance including Gitea, and renaming it here silently stops alerting
+for all of them.
+
+
+## Restore
+
+**Untested backups are not a control.** Rehearse this into a scratch location
+before you need it, and record the date you last did.
+
+### 1. Database
+
+Dumps live on the host at `/var/backups/nextcloud/<name>/db/<name>-YYYYMMDD.sql.gz`
+and on TrueNAS at `<remote_path>/_backup/db/`. TrueNAS in turn cloud-syncs
+`/mnt/glacier/skudakcloud` to Skudak's own iDrive e2 bucket, so a third copy
+exists there — but restoring from it means going through the TrueNAS console,
+not this host.
+
+Verify the dump before trusting it:
+
+```bash
+gzip -t <name>-YYYYMMDD.sql.gz
+gunzip -c <name>-YYYYMMDD.sql.gz | tail -c 512   # expect the completion trailer
+```
+
+Replay into the running database container. MariaDB/MySQL:
+
+```bash
+sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
+  | podman exec -i <db_container> sh -c \
+    "exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"'
+```
+
+Postgres dumps are taken with `--clean --if-exists --no-owner`, so they replay
+into an existing database:
+
+```bash
+sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
+  | podman exec -i <db_container> sh -c \
+    "exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"'
+```
+
+### 2. Data tree
+
+```bash
+# from TrueNAS
+rsync -az -e "ssh -i /etc/ssh/backup_keys/<name>" \
+  <ssh_user>@truenas.localdomain:<remote_path>/ <volumes>/<instance>/data/
+
+```
+
+Then fix ownership — the containers run as uid 33 inside a rootless userns:
+
+```bash
+sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 <volumes>/<instance>/data'
+```
+
+### 3. Reconcile
+
+```bash
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --on'
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ files:scan --all'
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --off'
+```
+
+A DB snapshot slightly **older** than the files degrades to "files the app has
+not indexed yet" and is repaired by `files:scan`. A DB snapshot **newer** than
+the files references blobs that were never backed up, which surfaces as broken
+shares and dead file entries — this is why the dump runs first.
+
+### 4. LibreSign-specific
+
+The signing CA lives in the data tree at
+`data/appdata_*/libresign/pki/<instance>_<n>_openssl/`, so a data-tree restore
+brings it back with everything else. After restoring, confirm it:
+
+```bash
+sudo -H -u podman bash -c 'cd; podman exec -u www-data skudak-cloud php occ libresign:configure:check'
+```
+
+Every check must report `success`. If `openssl-configure` reports an error, the
+`certificate_engine` / `config_path` app config is pointing somewhere without a
+CA — see the guarded generate task in `tasks/containers/skudak/cloud.yml`.
+**Do not** simply re-run `libresign:configure:openssl` on a restored instance
+without understanding why: it mints a *new* root CA and invalidates the trust
+chain on every document already signed under the old one.
+
+## LibreSign
+
+Deployed on `skudak-cloud` only. LibreSign 14.1.0 requires Nextcloud server
+`>=34.0.0,<35.0.0`, which the pinned `nextcloud:34.0.2-apache` satisfies. If the
+Nextcloud tag is bumped to 35, LibreSign must be held or upgraded in step — the
+two instances are pinned independently in `tasks/main.yml`, so `skudak-cloud`
+can lag `cloud` if needed.
+
+Dependency split, which drives what survives a container recreate:
+
+| Component | Location | Survives recreate? |
+|---|---|---|
+| Java (JRE 21), PDFtk, jSignPdf | `data/appdata_*/libresign/` | **Yes** — persisted volume |
+| Root CA / PKI | `data/appdata_*/libresign/pki/` | **Yes** — persisted volume |
+| poppler-utils, ghostscript | `/usr` in the image | **No** — reinstalled by Ansible each run |
+
+Certificate engine is **OpenSSL**, not CFSSL. CFSSL is the more common source of
+LibreSign setup failures and buys nothing at this scale.
+
+### Gotchas
+
+- **Do not pass `--ou`** to `libresign:configure:openssl`. LibreSign appends its
+  own `libresign-ca-id:...` entry to the OU field, and the combined value
+  overruns the 64-character ASN.1 limit for `organizationalUnitName`, failing
+  with `string too long`.
+- **A disabled app has no `occ` commands.** If `occ list | grep libresign`
+  returns nothing, the app is disabled, not missing — `occ app:list` will still
+  show it under `Disabled:`. This is what a Nextcloud major upgrade does to an
+  app it thinks is incompatible.
+- `PHP_MEMORY_LIMIT` must be raised above the 512M image default; signing fails
+  opaquely mid-operation otherwise.
+- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and
+  LibreSign warns that accented characters in signer names will be mangled
+  (LibreSign issue #4872).
@@ -110,6 +110,25 @@ bookstack_server_name_new: wiki.skudak.com
 cloud_skudak_server_name_new: cloud.skudak.com
 gitea_skudak_server_name: git.skudak.com

+# LibreSign root certificate authority identity for skudak-cloud. This is the
+# issuer name that appears on every signed document, so it must match the
+# entity's legal name -- it was previously generated as "Skudak Rennsport LLP",
+# the pre-rename name. Changing these values does NOT re-issue the CA on its
+# own; see the guarded generate task in containers/skudak/cloud.yml.
+# Skudak brand palette, mirroring ~/src/skudak/skudak-site/src/styles/variables.css.
+# --color-gray-900 for the mail header band and UI chrome; the white signature
+# logo is legible on it. --color-accent is spent only on the CTA button, and
+# lives in the skudakmail app rather than here since theming has no second
+# colour slot.
+theming_skudak_primary: "#0A0A0A"
+
+libresign_skudak_cert_cn: Skudak LLP
+libresign_skudak_cert_o: Skudak LLP
+libresign_skudak_cert_c: US
+libresign_skudak_cert_st: New Hampshire
+libresign_skudak_cert_l: Newbury
+
+
 # Legacy nginx/ModSecurity configuration removed - Caddy provides built-in security

 # Web server configuration (Caddy is the default)
@@ -194,3 +213,14 @@ podman_prune_users:
 # Keep 30 days of unused images so a rollback needs no rebuild or re-pull.
 podman_prune_until: 720h
 podman_prune_oncalendar: "Sun *-*-* 02:00:00"
+
+# Hardened CIFS options for the TrueNAS shares (see containers/home/photos.yml).
+# x-systemd.automount is what makes a failed mount recoverable without a human.
+cifs_mount_opts: >-
+  credentials=/etc/cifs/photos.creds,uid={{ podman_subuid.stdout }},gid={{ podman_subuid.stdout }},_netdev,nofail,soft,x-systemd.automount,x-systemd.mount-timeout=30,x-systemd.idle-timeout=600,vers=3.1.1,resilienthandles,echo_interval=10
+# How often the watchdog checks mount health and heals immich.
+cifs_watchdog_oncalendar: "*:0/5"
+cifs_watchdog_mounts:
+  - "{{ photos_path }}/storage"
+  - "{{ photos_path }}/immich"
+
@@ -0,0 +1,44 @@
+<?xml version="1.0"?>
+<info xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+      xsi:noNamespaceSchemaLocation="https://apps.nextcloud.com/schema/apps/info.xsd">
+    <id>skudakmail</id>
+    <name>Skudak Customisations</name>
+    <summary>Skudak-branded email templates and UI overrides for Nextcloud and LibreSign</summary>
+    <description><![CDATA[
+Restyles outgoing Nextcloud and LibreSign mail to match the Skudak design
+system at ~/src/skudak/skudak-site. Two supported extension points, no core
+patch and no LibreSign fork:
+
+1. `OCA\Skudakmail\Mail\SkudakEMailTemplate` extends Nextcloud's EMailTemplate
+   and is wired in via the `mail_template_class` system config value, which
+   Nextcloud checks in `lib/private/Mail/Mailer.php::createEMailTemplate()`.
+   It owns layout, typography, subject rewriting, button labels and the
+   footer LibreSign never adds.
+
+2. `OCA\Skudakmail\Listener\SkudakMailListener` listens on
+   `OCP\Mail\Events\BeforeMessageSent` to embed the wordmark as an inline
+   (cid:) MIME part, so the logo survives the remote-image blocking that
+   Apple Mail, Gmail and Outlook apply by default. This cannot be done from
+   the template class, which has no reference to the message.
+
+3. `OCA\Skudakmail\Listener\SkudakStyleListener` listens on
+   `OCP\AppFramework\Http\Events\BeforeTemplateRenderedEvent` and adds
+   css/libresign-mobile.css, which fixes the LibreSign public signing page
+   being clipped at the bottom on iOS Safari. Serving it from here rather
+   than patching LibreSign keeps the app's integrity signature intact and
+   survives app updates, which wipe the app directory.
+
+The app has no routes, no UI, no settings and no database tables. The id
+remains `skudakmail` for historical reasons -- it is referenced by the
+`mail_template_class` system config -- but its scope is Skudak-wide
+customisation, not mail alone.
+    ]]></description>
+    <version>1.0.0</version>
+    <licence>agpl</licence>
+    <author>Skudak LLP</author>
+    <namespace>Skudakmail</namespace>
+    <category>customization</category>
+    <dependencies>
+        <nextcloud min-version="34" max-version="34"/>
+    </dependencies>
+</info>
@@ -0,0 +1,40 @@
+/*
+ * Mobile fix for the LibreSign public signing page.
+ *
+ * PROBLEM: src/ExternalApp.vue sets `height: 100vh` on `html body #content`
+ * and again on `#app-sidebar` under `@media (max-width: 512px)`. iOS Safari
+ * resolves 100vh against the LARGE viewport -- as though the browser chrome
+ * were hidden -- so the element extends behind the bottom toolbar and the
+ * signing action bar is clipped off-screen. The built `external` chunk uses
+ * 100vh seven times and dvh/svh/safe-area zero times.
+ *
+ * WHY NOT safe-area-inset: the page's viewport meta is
+ * `width=device-width, initial-scale=1.0, minimum-scale=1.0` with no
+ * `viewport-fit=cover`, so env(safe-area-inset-bottom) resolves to 0 here.
+ *
+ * WHY dvh: the dynamic viewport unit tracks the chrome as it shows and hides,
+ * which is exactly the behaviour wanted. Browsers without dvh support drop the
+ * declaration entirely and keep LibreSign's own 100vh -- so this degrades to
+ * today's behaviour rather than to something broken. No @supports needed.
+ *
+ * SCOPING IS LOad-BEARING. `#content` and `#app-sidebar` are Nextcloud-wide
+ * IDs used throughout the authenticated UI. Every rule below is scoped to
+ * `#body-public` + `.app-public`, which the public signing page sets:
+ *     <body id="body-public" class="layout-base">
+ *     <div id="content" class="app-public" role="main">
+ * Widening these selectors would restyle the whole instance.
+ *
+ * UPSTREAM: patched at source in src/ExternalApp.vue (lines 34 and 46) and
+ * submitted to LibreSign. Once that lands and this instance runs a release
+ * containing it, this file can be deleted.
+ */
+
+#body-public #content.app-public {
+	height: 100dvh;
+}
+
+@media (max-width: 512px) {
+	#body-public #app-sidebar {
+		height: 100dvh;
+	}
+}
@@ -0,0 +1,41 @@
+<?php
+
+declare(strict_types=1);
+
+namespace OCA\Skudakmail\AppInfo;
+
+use OCA\Skudakmail\Listener\SkudakMailListener;
+use OCA\Skudakmail\Listener\SkudakStyleListener;
+use OCP\AppFramework\App;
+use OCP\AppFramework\Bootstrap\IBootContext;
+use OCP\AppFramework\Bootstrap\IBootstrap;
+use OCP\AppFramework\Bootstrap\IRegistrationContext;
+use OCP\AppFramework\Http\Events\BeforeTemplateRenderedEvent;
+use OCP\Mail\Events\BeforeMessageSent;
+
+class Application extends App implements IBootstrap {
+	public const APP_ID = 'skudakmail';
+
+	public function __construct(array $urlParams = []) {
+		parent::__construct(self::APP_ID, $urlParams);
+	}
+
+	public function register(IRegistrationContext $context): void {
+		// BeforeMessageSent fires in Mailer::send() (lib/private/Mail/Mailer.php:186),
+		// AFTER useTemplate() has flattened the template into subject/plain/html on
+		// the message, and BEFORE setRecipients() and the Symfony transport. That
+		// window is the only place an inline (cid:) logo can be attached -- see the
+		// listener for why the template class alone cannot do it.
+		$context->registerEventListener(BeforeMessageSent::class, SkudakMailListener::class);
+
+		// BeforeTemplateRenderedEvent is dispatched from
+		// lib/private/AppFramework/Middleware/AdditionalScriptsMiddleware.php:35 and
+		// lib/private/Template/TemplateManager.php:82 -- the latter covers public
+		// (unauthenticated) pages, which is the case that matters here since the
+		// LibreSign signing page is a #[PublicPage].
+		$context->registerEventListener(BeforeTemplateRenderedEvent::class, SkudakStyleListener::class);
+	}
+
+	public function boot(IBootContext $context): void {
+	}
+}
@@ -0,0 +1,121 @@
+<?php
+
+declare(strict_types=1);
+
+/**
+ * Embeds the Skudak wordmark as an inline (cid:) MIME part.
+ *
+ * WHY A LISTENER AND NOT THE TEMPLATE CLASS: Apple Mail, Gmail and Outlook all
+ * block remote images by default, and Apple Mail draws its own placeholder box
+ * rather than styled alt text -- so no amount of styling in the HTML rescues a
+ * remote <img>. The fix is a cid: reference backed by an inline MIME part, and
+ * that part must be attached to the MESSAGE. An IEMailTemplate subclass has no
+ * reference to the message, so it physically cannot do this; the template emits
+ * the <img>, this listener supplies the bytes and rewrites the src.
+ *
+ * BeforeMessageSent is the sanctioned hook -- "Emitted before a system mail is
+ * sent. It can be used to alter the message." (lib/public/Mail/Events/
+ * BeforeMessageSent.php). It fires at lib/private/Mail/Mailer.php:186, after
+ * useTemplate() has already rendered subject/plain/html onto the message and
+ * before setRecipients() and the transport, so a body rewrite here takes
+ * effect. No core patch, no LibreSign fork.
+ *
+ * FAILURE POSTURE: every step is defensive. If the asset is missing, the body
+ * is not ours, or anything throws, the listener leaves the message untouched
+ * and mail still goes out with a remote <img> -- degraded, never blocked. Mail
+ * that carries signature requests must not fail to send because branding
+ * broke.
+ */
+
+namespace OCA\Skudakmail\Listener;
+
+use OC\Mail\Message;
+use OCP\EventDispatcher\Event;
+use OCP\EventDispatcher\IEventListener;
+use OCP\Mail\Events\BeforeMessageSent;
+use Psr\Log\LoggerInterface;
+
+/** @template-implements IEventListener<BeforeMessageSent> */
+class SkudakMailListener implements IEventListener {
+	/** Must match SkudakEMailTemplate::LOGO_PATH. */
+	private const LOGO_PATH_FRAGMENT = '/custom_apps/skudakmail/img/skudak-wordmark.png';
+
+	/** Content-ID. Symfony emits this as <skudak-wordmark.png>. */
+	private const CID = 'skudak-wordmark.png';
+
+	public function __construct(
+		private LoggerInterface $logger,
+	) {
+	}
+
+	public function handle(Event $event): void {
+		if (!$event instanceof BeforeMessageSent) {
+			return;
+		}
+
+		try {
+			$this->embedWordmark($event->getMessage());
+		} catch (\Throwable $e) {
+			// Never let branding break delivery of a signature request.
+			$this->logger->warning('skudakmail: inline logo embed skipped', [
+				'exception' => $e,
+			]);
+		}
+	}
+
+	private function embedWordmark(\OCP\Mail\IMessage $message): void {
+		// Mailer::send() guards `instanceof Message` before dispatching this
+		// event, so the concrete type is guaranteed -- but getSymfonyEmail()
+		// is not on the interface, so narrow explicitly rather than assume.
+		if (!$message instanceof Message) {
+			return;
+		}
+
+		$email = $message->getSymfonyEmail();
+		$html = $email->getHtmlBody();
+		if (!is_string($html) || $html === '') {
+			return;
+		}
+
+		// Only touch mail that actually renders our wordmark. Anything else --
+		// password resets, share notifications, other apps -- passes through.
+		if (!str_contains($html, self::LOGO_PATH_FRAGMENT)) {
+			return;
+		}
+
+		$asset = $this->assetPath();
+		if ($asset === null) {
+			return;
+		}
+
+		$bytes = @file_get_contents($asset);
+		if ($bytes === false || $bytes === '') {
+			return;
+		}
+
+		// Rewrite the absolute URL to a cid: reference. Matched on the path
+		// fragment with an optional query string so a cachebuster or a change
+		// of host still resolves.
+		$rewritten = preg_replace(
+			'#https?://[^"\']*' . preg_quote(self::LOGO_PATH_FRAGMENT, '#') . '(\?[^"\']*)?#',
+			'cid:' . self::CID,
+			$html,
+		);
+
+		if (!is_string($rewritten) || $rewritten === $html) {
+			return;
+		}
+
+		$email->embed($bytes, self::CID, 'image/png');
+		$message->setHtmlBody($rewritten);
+	}
+
+	/**
+	 * Resolves img/skudak-wordmark.png relative to this file, so the app works
+	 * from whatever apps directory Nextcloud has it in.
+	 */
+	private function assetPath(): ?string {
+		$path = dirname(__DIR__, 2) . '/img/skudak-wordmark.png';
+		return is_readable($path) ? $path : null;
+	}
+}
@@ -0,0 +1,51 @@
+<?php
+
+declare(strict_types=1);
+
+/**
+ * Injects Skudak's CSS overrides into rendered Nextcloud pages.
+ *
+ * Currently one override: the LibreSign public signing page clips its bottom
+ * action bar on iOS Safari, because ExternalApp.vue sizes #content to 100vh and
+ * Safari resolves that against the large viewport (chrome hidden). See
+ * css/libresign-mobile.css for the full reasoning.
+ *
+ * WHY A LISTENER RATHER THAN PATCHING LIBRESIGN: an app-store app carries
+ * appinfo/signature.json, so editing a single byte of it raises INVALID_HASH in
+ * the admin security check, and an app update wipes the directory outright
+ * (Installer::downloadApp() calls Files::rmdirr on it). A stylesheet served
+ * from our own app survives both, and survives Nextcloud upgrades.
+ *
+ * The stylesheet itself is tightly scoped to #body-public / .app-public. This
+ * listener is deliberately NOT scoped further -- adding a stylesheet is
+ * idempotent and cheap, and gating on which app is rendering would couple this
+ * to LibreSign's route structure for no benefit. The CSS decides where it
+ * applies; this only decides that it is available.
+ */
+
+namespace OCA\Skudakmail\Listener;
+
+use OCA\Skudakmail\AppInfo\Application;
+use OCP\AppFramework\Http\Events\BeforeTemplateRenderedEvent;
+use OCP\EventDispatcher\Event;
+use OCP\EventDispatcher\IEventListener;
+use OCP\Util;
+
+/** @template-implements IEventListener<BeforeTemplateRenderedEvent> */
+class SkudakStyleListener implements IEventListener {
+	public function handle(Event $event): void {
+		if (!$event instanceof BeforeTemplateRenderedEvent) {
+			return;
+		}
+
+		// Never let a styling concern break page rendering. A signing page that
+		// loads unstyled is recoverable; one that 500s is not.
+		try {
+			Util::addStyle(Application::APP_ID, 'libresign-mobile');
+		} catch (\Throwable $e) {
+			// Intentionally swallowed -- no logger dependency is worth adding
+			// for a stylesheet, and a failure here has no user-visible effect
+			// beyond the override not applying.
+		}
+	}
+}
@@ -0,0 +1,417 @@
+<?php
+
+declare(strict_types=1);
+
+/**
+ * Skudak-branded email template.
+ *
+ * Wired in via the `mail_template_class` system config value, which Nextcloud
+ * checks in lib/private/Mail/Mailer.php::createEMailTemplate(). That is a
+ * supported extension point -- core is not patched, so Nextcloud upgrades do
+ * not clobber this.
+ *
+ * WHY THIS EXISTS AT ALL: LibreSign's outgoing mail is generic open-source
+ * boilerplate -- subject "LibreSign: There is a file for you to sign", heading
+ * "File to sign", button "Sign »filename«", and NO footer whatsoever (it never
+ * calls addFooter(); verified: zero hits for addFooter in custom_apps/libresign).
+ * That mail carries partnership instruments to signers, so it needs to read as
+ * an official Skudak LLP communication.
+ *
+ * DESIGN INTENT (mirrors ~/src/skudak/skudak-site/src/styles/variables.css):
+ *   - Light ground, near-black text, NO coloured header band. skudak.com is
+ *     --color-white #FAFAFA with --color-gray-900 #0A0A0A text; transactional
+ *     mail from Stripe/Linear/DocuSign is likewise restrained. A band also
+ *     leaves an ugly empty slab when the logo is blocked (see LOGO note).
+ *   - --color-accent #2563EB on the CTA button only. That is the one place the
+ *     site spends colour, so it is the one place this template does.
+ *   - Inter, matching --font-sans, with the stock stack as fallback.
+ *
+ * LOGO: served from this app's own img/ directory rather than the theming app.
+ * Two reasons. (1) The theming logo is white-on-transparent because the web UI
+ * and login page are dark; a white mark is invisible on this template's white
+ * ground. (2) Decoupling means restyling mail can never disturb the web UI.
+ * /custom_apps/<app>/img/<file> is served publicly without auth (verified).
+ *
+ * Note that remote images are blocked by default in Apple Mail, Gmail and
+ * Outlook, and Apple Mail renders its own placeholder box rather than styled
+ * alt text -- so alt styling cannot rescue it. Surviving that requires a CID
+ * inline part via IMessage::attachInline(), which lives on the MESSAGE and is
+ * unreachable from a template subclass. Mitigated instead by dropping the
+ * band: a blocked logo now leaves plain white space, not a black slab.
+ *
+ * IMPLEMENTATION NOTE: font restyling is done by string-substitution against
+ * the PARENT's own markup rather than by redefining it. Those properties are
+ * large inline-CSS blobs with positional sprintf placeholders; copying them
+ * wholesale would mean re-auditing every placeholder on every upgrade, and a
+ * mismatch renders broken mail. Substitution degrades safely -- if upstream
+ * changes markup the replacements no-op and mail still sends, just unstyled.
+ * The header IS replaced wholesale, deliberately, because "no band" cannot be
+ * expressed as a substitution; its placeholder order is documented at its
+ * definition and must be kept in sync with upstream.
+ */
+
+namespace OCA\Skudakmail\Mail;
+
+use OC\Mail\EMailTemplate;
+
+class SkudakEMailTemplate extends EMailTemplate {
+	/** Stock Nextcloud font stack, replaced wholesale. Must match exactly. */
+	private const STOCK_FONTS = "-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',Arial,sans-serif";
+
+	/** --font-sans, with the stock stack retained as fallback. */
+	private const SKUDAK_FONTS = "Inter,-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Oxygen-Sans,Ubuntu,Cantarell,'Helvetica Neue',Arial,sans-serif";
+
+	private const ACCENT = '#2563EB';       // --color-accent
+	private const ON_ACCENT = '#FAFAFA';    // --color-white
+	private const INK = '#0A0A0A';          // --color-gray-900
+	private const MUTED = '#525252';        // --color-gray-500
+	private const FAINT = '#A3A3A3';        // --color-gray-300
+	private const RULE = '#E5E5E5';         // --color-gray-100
+
+	private const ENTITY = 'Skudak LLP';
+	private const SITE = 'https://skudak.com';
+	private const LOGO_PATH = '/custom_apps/skudakmail/img/skudak-wordmark.png';
+
+	/** Displayed width in px. The asset is 600px wide for retina. */
+	private const LOGO_DISPLAY_WIDTH = 190;
+
+	/**
+	 * LibreSign's l10n wraps document names in German guillemets -- "Sign
+	 * »contract«" -- regardless of locale. Mapped to US curly quotes, matching
+	 * the ``...'' convention in the LaTeX document templates.
+	 */
+	private const QUOTE_MAP = ['»' => "\u{201C}", '«' => "\u{201D}"];
+
+	/**
+	 * LibreSign subject -> Skudak subject. Keys are the exact English msgids
+	 * from custom_apps/libresign/lib/Service/MailService.php (lines 51, 87,
+	 * 121, 150, 172). Anything unmatched passes through untouched, so an
+	 * upstream string change degrades to the original subject rather than a
+	 * blank one.
+	 */
+	private const SUBJECT_MAP = [
+		'LibreSign: There is a file for you to sign' => 'Document for your signature',
+		'LibreSign: Changes into a file for you to sign' => 'Updated document for your signature',
+		'LibreSign: A file has been signed' => 'A document has been signed',
+		'LibreSign: A signature request has been canceled' => 'Signature request cancelled',
+		'LibreSign: Code to sign file' => 'Your signing verification code',
+	];
+
+	/**
+	 * LibreSign heading -> Skudak heading. Exact English msgids from
+	 * MailService.php lines 53/89, 123, 152.
+	 */
+	private const HEADING_MAP = [
+		'File to sign' => 'Review and sign',
+		'File signed' => 'Document signed',
+		'Signature request canceled' => 'Signature request cancelled',
+	];
+
+	/**
+	 * LibreSign body copy -> Skudak body copy (MailService.php lines 60, 96,
+	 * 174). Only the strings with NO %s interpolation are mapped; the two that
+	 * carry a name or filename (lines 125, 154) arrive already substituted and
+	 * so cannot be matched exactly -- they pass through unchanged.
+	 */
+	private const BODY_MAP = [
+		'There is a document for you to sign. Access the link below:'
+			=> 'Skudak LLP has sent you a document that requires your signature. Review it and sign using the link below.',
+		'Changes have been made in a file that you have to sign. Access the link below:'
+			=> 'A document awaiting your signature has been updated by Skudak LLP. Review the current version and sign using the link below.',
+		'Use this code to sign the document:'
+			=> 'Use this verification code to complete your signature:',
+	];
+
+	/**
+	 * Template properties carrying the font stack. Listed explicitly rather
+	 * than discovered reflectively so an upstream rename fails loudly in
+	 * testing instead of silently skipping a block.
+	 */
+	private const STYLED_PARTS = [
+		'head', 'tail', 'heading', 'bodyBegin', 'bodyText',
+		'listBegin', 'listItem', 'listEnd', 'buttonGroup', 'button',
+		'bodyEnd', 'footer',
+	];
+
+	/**
+	 * Own flag, deliberately NOT the parent's $footerAdded.
+	 *
+	 * Message::useTemplate() (lib/private/Mail/Message.php:289-296) calls
+	 * renderText() at :291 BEFORE renderHtml() at :293, and renderText() sets
+	 * $footerAdded = true. Guarding footer injection on !$footerAdded therefore
+	 * never fires on the real send path -- the footer silently vanished from
+	 * every mail while a renderHtml()-only test passed. Both renderers below
+	 * call inject() and this flag makes the second call inert.
+	 */
+	private bool $skudakFooterInjected = false;
+
+	public function __construct(
+		\OCP\Defaults $themingDefaults,
+		\OCP\IURLGenerator $urlGenerator,
+		\OCP\L10N\IFactory $l10nFactory,
+		?int $logoWidth,
+		?int $logoHeight,
+		string $emailId,
+		array $data,
+	) {
+		$this->applySkudakStyling();
+
+		// Must run AFTER the substitutions: the parent constructor copies
+		// $this->head into $htmlBody as its first act, so restyling head
+		// afterwards would leave the already-emitted copy untouched.
+		parent::__construct(
+			$themingDefaults,
+			$urlGenerator,
+			$l10nFactory,
+			$logoWidth,
+			$logoHeight,
+			$emailId,
+			$data,
+		);
+	}
+
+	private function applySkudakStyling(): void {
+		foreach (self::STYLED_PARTS as $part) {
+			if (!property_exists($this, $part)) {
+				continue;
+			}
+			$this->$part = str_replace(self::STOCK_FONTS, self::SKUDAK_FONTS, $this->$part);
+		}
+
+		// Site headings are --font-weight-light with tightened tracking.
+		$this->heading = str_replace(
+			'font-size:24px;font-weight:400',
+			'font-size:26px;font-weight:300;letter-spacing:-0.02em',
+			$this->heading,
+		);
+	}
+
+	/**
+	 * Rewrites LibreSign's subjects. Called by LibreSign on the TEMPLATE
+	 * (MailService.php:51 etc.), not on the message, which is what makes this
+	 * interceptable at all -- Message::useTemplate() later pulls the result via
+	 * renderSubject(). Prefixed with the entity so the sender is unambiguous in
+	 * an inbox list.
+	 */
+	public function setSubject(string $subject): void {
+		$mapped = self::SUBJECT_MAP[$subject] ?? null;
+		parent::setSubject(
+			$mapped === null ? $subject : self::ENTITY . ' — ' . $mapped,
+		);
+	}
+
+	/**
+	 * Replaces the stock header wholesale: no coloured band, wordmark centred
+	 * on white.
+	 *
+	 * Does NOT use the parent's $header property or its placeholder order --
+	 * this is independent markup, so upstream changes to $header cannot break
+	 * it (and equally cannot improve it). $logoWidth/$logoHeight from the
+	 * Mailer are ignored on purpose: they are clamped to MAX_LOGO_SIZE = 105
+	 * (lib/private/Mail/Mailer.php:60), which is too small for a wordmark to
+	 * be legible.
+	 */
+	public function addHeader(): void {
+		if ($this->headerAdded) {
+			return;
+		}
+		$this->headerAdded = true;
+
+		$logoUrl = $this->urlGenerator->getAbsoluteURL(self::LOGO_PATH);
+		$alt = htmlspecialchars(self::ENTITY, ENT_QUOTES, 'UTF-8');
+		$w = self::LOGO_DISPLAY_WIDTH;
+		$fonts = self::SKUDAK_FONTS;
+		$ink = self::INK;
+
+		$this->htmlBody .= <<<HTML
+<table align="center" style="border-collapse:collapse;border-spacing:0;margin:0 auto;padding:0;text-align:left;vertical-align:top;width:100%">
+	<tbody><tr style="padding:0;text-align:left;vertical-align:top">
+		<td align="center" style="border-collapse:collapse!important;margin:0;padding:40px 30px 28px 30px;text-align:center;vertical-align:top">
+			<img src="{$logoUrl}" alt="{$alt}" width="{$w}" style="-ms-interpolation-mode:bicubic;border:0;clear:both;display:block;margin:0 auto;outline:0;text-decoration:none;width:{$w}px;max-width:{$w}px;height:auto;color:{$ink};font-family:{$fonts};font-size:22px;font-weight:300;letter-spacing:-0.02em"/>
+		</td>
+	</tr></tbody>
+</table>
+HTML;
+	}
+
+	/**
+	 * Both renderers inject the footer -- see $skudakFooterInjected.
+	 *
+	 * Mirrors the parent's own guard structure (renderHtml at
+	 * lib/private/Mail/EMailTemplate.php:643, renderText at :656): close the
+	 * body, append $tail, flip $footerAdded. The Skudak block goes in before
+	 * $tail.
+	 */
+	public function renderHtml(): string {
+		$this->injectSkudakFooter();
+		return parent::renderHtml();
+	}
+
+	public function renderText(): string {
+		$this->injectSkudakFooter();
+		return parent::renderText();
+	}
+
+	private function injectSkudakFooter(): void {
+		if ($this->skudakFooterInjected || $this->footerAdded) {
+			return;
+		}
+		$this->skudakFooterInjected = true;
+
+		// Close the body ourselves so the footer lands INSIDE the layout
+		// rather than after it. The parent's render methods are then a no-op
+		// for body closing and only append $tail.
+		$this->ensureBodyIsClosed();
+		$this->htmlBody .= $this->skudakFooterHtml();
+		$this->plainBody .= $this->skudakFooterText();
+	}
+
+	private function skudakFooterHtml(): string {
+		$year = date('Y');
+		$entity = htmlspecialchars(self::ENTITY, ENT_QUOTES, 'UTF-8');
+		$fonts = self::SKUDAK_FONTS;
+		$site = self::SITE;
+		[$muted, $faint, $rule, $ink] = [self::MUTED, self::FAINT, self::RULE, self::INK];
+
+		// Table-based and fully inline-styled: <style> blocks, flex and grid
+		// are stripped or unsupported across Outlook and most webmail.
+		return <<<HTML
+<table align="center" style="border-collapse:collapse;border-spacing:0;margin:0 auto;padding:0;text-align:left;vertical-align:top;width:100%">
+	<tbody><tr style="padding:0;text-align:left;vertical-align:top">
+		<td align="center" style="border-collapse:collapse!important;margin:0;padding:0 30px 44px 30px;text-align:center;vertical-align:top">
+			<table align="center" style="border-collapse:collapse;border-spacing:0;margin:0 auto;padding:0;text-align:center;width:100%;max-width:550px">
+				<tbody>
+				<tr><td style="border-collapse:collapse!important;border-top:1px solid {$rule};font-size:0;line-height:0;height:1px;margin:0;padding:0">&#xA0;</td></tr>
+				<tr><td align="center" style="border-collapse:collapse!important;color:{$muted};font-family:{$fonts};font-size:13px;font-weight:400;line-height:1.6;margin:0;padding:22px 0 0 0;text-align:center">
+					This is an official document-signing request from <strong style="color:{$ink};font-weight:600">{$entity}</strong>.<br/>
+					Nothing is signed unless you open the document and complete it yourself. If you were not expecting this, you can safely ignore it.
+				</td></tr>
+				<tr><td align="center" style="border-collapse:collapse!important;color:{$muted};font-family:{$fonts};font-size:13px;font-weight:400;line-height:1.6;margin:0;padding:16px 0 0 0;text-align:center">
+					<a href="{$site}/privacy" style="color:{$muted};text-decoration:underline">Privacy Policy</a>
+					&#160;&#183;&#160;
+					<a href="{$site}/terms" style="color:{$muted};text-decoration:underline">Terms of Use</a>
+					&#160;&#183;&#160;
+					<a href="{$site}" style="color:{$muted};text-decoration:underline">skudak.com</a>
+				</td></tr>
+				<tr><td align="center" style="border-collapse:collapse!important;color:{$faint};font-family:{$fonts};font-size:12px;font-weight:400;line-height:1.6;margin:0;padding:16px 0 0 0;text-align:center">
+					&copy; {$year} {$entity}. All rights reserved.<br/>
+					Automated message &mdash; please do not reply to this address.
+				</td></tr>
+				</tbody>
+			</table>
+		</td>
+	</tr></tbody>
+</table>
+HTML;
+	}
+
+	private function skudakFooterText(): string {
+		$year = date('Y');
+		$entity = self::ENTITY;
+		$site = self::SITE;
+
+		return <<<TEXT
+
+--
+This is an official document-signing request from {$entity}.
+Nothing is signed unless you open the document and complete it yourself.
+If you were not expecting this, you can safely ignore it.
+
+Privacy Policy: {$site}/privacy
+Terms of Use:   {$site}/terms
+
+© {$year} {$entity}. All rights reserved.
+Automated message — please do not reply to this address.
+
+TEXT;
+	}
+
+	private function tidyQuotes(string $text): string {
+		return strtr($text, self::QUOTE_MAP);
+	}
+
+	// Signatures below mirror the parent EXACTLY. $plainTitle/$plainText are
+	// deliberately untyped there (they accept string|bool -- false suppresses
+	// the plain-text variant), and narrowing a parameter type in an override
+	// is a fatal error in PHP.
+	public function addHeading(string $title, $plainTitle = ''): void {
+		$mapped = self::HEADING_MAP[$title] ?? $this->tidyQuotes($title);
+		parent::addHeading(
+			$mapped,
+			is_string($plainTitle) && $plainTitle !== ''
+				? (self::HEADING_MAP[$plainTitle] ?? $this->tidyQuotes($plainTitle))
+				: $plainTitle,
+		);
+	}
+
+	public function addBodyText(string $text, $plainText = ''): void {
+		$mapped = self::BODY_MAP[$text] ?? $this->tidyQuotes($text);
+		parent::addBodyText(
+			$mapped,
+			is_string($plainText) && $plainText !== ''
+				? (self::BODY_MAP[$plainText] ?? $this->tidyQuotes($plainText))
+				: $plainText,
+		);
+	}
+
+	/**
+	 * Reimplemented for two reasons: the accent colour, and a fixed label.
+	 *
+	 * LibreSign builds "Sign »%s«" with the raw filename
+	 * (MailService.php:64,100). Real documents here are named things like
+	 * amendment-001-partner-compensation, which makes an ungainly button and
+	 * leaks the document name to anyone who sees the inbox preview. Replaced
+	 * with a fixed call to action; the document is identified on the landing
+	 * page behind the link.
+	 *
+	 * Mirrors the parent's vsprintf argument order exactly:
+	 * [$color, $color, $url, $color, $textColor, $textColor, $text].
+	 * Kept in sync with parent::addBodyButton() -- if that changes upstream,
+	 * this needs revisiting.
+	 */
+	public function addBodyButton(string $text, string $url, $plainText = ''): void {
+		if ($this->footerAdded) {
+			return;
+		}
+
+		$this->ensureBodyIsOpened();
+		$this->ensureBodyListClosed();
+
+		$label = $this->buttonLabelFor($text);
+		if ($plainText === '') {
+			$plainText = $label;
+		} elseif (is_string($plainText)) {
+			$plainText = $this->tidyQuotes($plainText);
+		}
+
+		$this->htmlBody .= vsprintf($this->button, [
+			self::ACCENT,
+			self::ACCENT,
+			$url,
+			self::ACCENT,
+			self::ON_ACCENT,
+			self::ON_ACCENT,
+			htmlspecialchars($label, ENT_QUOTES, 'UTF-8'),
+		]);
+
+		if ($plainText !== false) {
+			$this->plainBody .= $plainText . ': ';
+		}
+		$this->plainBody .= $url . PHP_EOL;
+	}
+
+	/**
+	 * Maps LibreSign's filename-bearing labels onto fixed calls to action.
+	 * Matched on the stable leading verb rather than the whole string, since
+	 * the tail is a filename. Unknown labels pass through with quotes tidied.
+	 */
+	private function buttonLabelFor(string $text): string {
+		if (str_starts_with($text, 'Sign ')) {
+			return 'Review document';
+		}
+		if (str_starts_with($text, 'View signed file')) {
+			return 'View signed document';
+		}
+		return $this->tidyQuotes($text);
+	}
+}
@@ -18,6 +18,18 @@
    mode: 0600
    setype: ssh_home_t

+# A direct host-to-S3 stage was added here and then removed. Offsite already
+# happens: the TrueNAS rsync below feeds /mnt/glacier/skudakcloud, and a
+# TrueNAS cloud-sync task pushes that to Skudak's own iDrive e2 bucket. A
+# second, direct push would have written the same data into the same bucket
+# twice. If offsite is ever moved onto this host, it should REPLACE the rsync
+# rather than run alongside it.
+- name: remove obsolete backup S3 credentials
+  become: true
+  ansible.builtin.file:
+    path: "/etc/backup_s3/{{ backup_name }}"
+    state: absent
+
 - name: template {{ backup_name }} backup script
  become: true
  ansible.builtin.template:
@@ -17,13 +17,56 @@
 - name: flush handlers
  ansible.builtin.meta: flush_handlers

+- name: create cifs credentials directory
+  become: true
+  ansible.builtin.file:
+    path: /etc/cifs
+    state: directory
+    owner: root
+    group: root
+    mode: 0700
+
+# /etc/fstab is 0644 by design, so an inline `password=` is readable by every
+# local user. Keep the credential in a 0600 file and reference it instead.
+- name: deploy photos cifs credentials
+  become: true
+  ansible.builtin.template:
+    src: cifs-credentials.j2
+    dest: /etc/cifs/photos.creds
+    owner: root
+    group: root
+    mode: 0600
+  vars:
+    cifs_username: photos
+    cifs_password: "{{ photos_cifs_pass }}"
+  no_log: true
+
+# These mounts previously failed permanently whenever TrueNAS was power-cycled:
+# a systemd .mount unit does NOT retry after a failed attempt, so the share came
+# back and nothing remounted, leaving immich serving an empty library while its
+# database still referenced 14k assets.
+#
+#   x-systemd.automount  the actual fix -- any ACCESS to the path re-attempts the
+#                        mount, so a failure stops being terminal
+#   _netdev / nofail     order after network-online; a dead NAS must not block boot
+#   soft                 fail I/O with an error instead of blocking forever, so
+#                        immich-server can be restarted during an outage rather
+#                        than wedging in uninterruptible sleep. Accepted trade-off:
+#                        a write interrupted mid-flight fails and is retried.
+#   idle-timeout         unmount when unused, which clears stale handles instead
+#                        of nursing a half-dead connection
+#   resilienthandles     SMB3 rides out brief server blips transparently
+#
+# See also cifs-watchdog.sh.j2, which restarts immich-server when a mount that
+# was unhealthy becomes healthy again -- the automount restores the FILESYSTEM,
+# but the container still holds the old, empty view until it is bounced.
 - name: mount photos cifs
  become: true
  ansible.posix.mount:
    src: "{{ photos_cifs_src }}"
    path: "{{ photos_path }}/storage"
    fstype: cifs
-    opts: "username=photos,password={{ photos_cifs_pass }},uid={{ podman_subuid.stdout }},gid={{ podman_subuid.stdout }}"
+    opts: "{{ cifs_mount_opts }}"
    state: mounted

 - name: mount immich cifs
@@ -32,9 +75,23 @@
    src: "{{ immich_cifs_src }}"
    path: "{{ photos_path }}/immich"
    fstype: cifs
-    opts: "username=photos,password={{ photos_cifs_pass }},uid={{ podman_subuid.stdout }},gid={{ podman_subuid.stdout }}"
+    opts: "{{ cifs_mount_opts }}"
    state: mounted

+# systemd-fstab-generator creates the .automount unit from x-systemd.automount,
+# but ansible.posix.mount mounts the path directly and never starts it, leaving
+# the on-access trigger INACTIVE -- so a dropped mount stayed dropped, which is
+# the exact failure this work exists to fix. Enable it explicitly.
+- name: enable cifs automount units
+  become: true
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+    daemon_reload: true
+  loop: "{{ cifs_watchdog_mounts | map('regex_replace', '^/', '') | map('regex_replace', '/', '-') | map('regex_replace', '$', '.automount') | list }}"
+  failed_when: false
+
 - import_tasks: podman/podman-check.yml
  vars:
    container_name: immich-machine-learning
@@ -25,15 +25,17 @@
 - name: flush handlers
  ansible.builtin.meta: flush_handlers

- name: copy skudak cloud libresign setup script
+# The former libresign-setup.sh before-starting hook re-ran
+# `occ libresign:install --java/--pdftk/--jsignpdf` on every container start,
+# with every line ending in `|| echo`, so six months of failures logged
+# nothing. Those binaries live under data/appdata_*/libresign, which IS a
+# persisted volume, so they only ever needed installing once. Installation and
+# verification are now explicit Ansible tasks below that actually fail.
+- name: remove obsolete skudak cloud libresign setup hook
  become: true
-  ansible.builtin.template:
-    src: nextcloud/libresign-setup.sh.j2
-    dest: "{{ cloud_skudak_path }}/scripts/libresign-setup.sh"
-    owner: "{{ podman_subuid.stdout }}"
-    group: "{{ podman_subuid.stdout }}"
-    mode: 0755
-  notify: restorecon podman
+  ansible.builtin.file:
+    path: "{{ cloud_skudak_path }}/scripts/libresign-setup.sh"
+    state: absent

 - import_tasks: podman/podman-check.yml
  vars:
@@ -63,6 +65,90 @@
  vars:
    container_name: skudak-cloud-db

+# ---------------------------------------------------------------------------
+# Redis: Nextcloud distributed cache + transactional file locking.
+#
+# Without it, memcache.locking is unset and Nextcloud falls back to
+# DBLockingProvider (lib/private/Server.php:977) -- every file lock becomes a
+# MariaDB write against oc_file_locks. A single directory PROPFIND takes dozens
+# of locks and two desktop sync clients issue them continuously, which is the
+# contention behind the intermittent multi-second stalls.
+#
+# Also fixes a second problem: memcache.local is APCu, which is PER-PROCESS,
+# and Apache here runs mpm_prefork -- so every child holds its own cold cache.
+# A distributed cache is shared across all of them.
+#
+# MUST be created before skudak-cloud below. The Nextcloud entrypoint writes
+# its redis config on start; if the host does not resolve at that moment the
+# instance comes up pointing at nothing.
+- name: create skudak cloud redis config directory
+  become: true
+  ansible.builtin.file:
+    path: "{{ cloud_skudak_path }}/redis"
+    state: directory
+    owner: "{{ podman_subuid.stdout }}"
+    group: "{{ podman_subuid.stdout }}"
+    mode: 0755
+  notify: restorecon podman
+
+- name: template skudak cloud redis config
+  become: true
+  ansible.builtin.template:
+    src: nextcloud/redis-skudak.conf.j2
+    dest: "{{ cloud_skudak_path }}/redis/redis.conf"
+    owner: "{{ podman_subuid.stdout }}"
+    group: "{{ podman_subuid.stdout }}"
+    mode: 0640
+  notify: restorecon podman
+  no_log: true
+
+- name: flush handlers
+  ansible.builtin.meta: flush_handlers
+
+# The redis:alpine image runs redis-server as uid 999 / gid 1000, NOT root, and
+# the config is mounted :ro so the image's own entrypoint cannot chown it --
+# it logs "cannot change owner ... Read-only file system" and then dies with
+# "Fatal error, can't open config file: Permission denied", crash-looping.
+# Nextcloud, already pointed at redis by then, answers HTTP 500.
+#
+# Same idiom as the `podman unshare chown -R 33:33` for www-data above: map the
+# in-container uid through the rootless userns. 0640 owned by 999:1000 keeps
+# the password unreadable to other users on the host while letting redis read
+# it -- which is the entire reason for using a file over --requirepass.
+- name: unshare chown the skudak redis config to the redis uid
+  become: true
+  become_user: "{{ podman_user }}"
+  changed_when: false
+  ansible.builtin.command: >
+    podman unshare chown 999:1000 {{ cloud_skudak_path }}/redis/redis.conf
+
+- import_tasks: podman/podman-check.yml
+  vars:
+    container_name: skudak-cloud-redis
+    container_image: "{{ redis_image }}"
+
+- name: create skudak-cloud-redis container
+  become: true
+  become_user: "{{ podman_user }}"
+  containers.podman.podman_container:
+    name: skudak-cloud-redis
+    image: "{{ redis_image }}"
+    restart_policy: on-failure:3
+    log_driver: journald
+    network:
+      - shared
+    # No `ports:` -- deliberately unpublished. Service discovery is by
+    # container name over `shared`, the same way MYSQL_HOST reaches
+    # skudak-cloud-db.
+    volumes:
+      - "{{ cloud_skudak_path }}/redis/redis.conf:/etc/redis/redis.conf:ro"
+    command: redis-server /etc/redis/redis.conf
+
+- name: create systemd startup job for skudak-cloud-redis
+  include_tasks: podman/systemd-generate.yml
+  vars:
+    container_name: skudak-cloud-redis
+
 - import_tasks: podman/podman-check.yml
  vars:
    container_name: skudak-cloud
@@ -83,11 +169,34 @@
      MYSQL_DATABASE: skucloud
      MYSQL_HOST: skudak-cloud-db
      MYSQL_USER: skucloud
+      # LibreSign signs PDFs in-process; the image default of 512M is not
+      # enough and manifests as an opaque failure mid-signature.
+      PHP_MEMORY_LIMIT: 1024M
+      PHP_UPLOAD_LIMIT: 512M
+      # Without these the JVM comes up as ANSI_X3.4-1968 and LibreSign's
+      # config check warns that accented characters in signer names will be
+      # mangled. See LibreSign issue #4872.
+      LC_ALL: C.UTF-8
+      LANG: C.UTF-8
+      # These three env vars are the WHOLE redis wiring. The image ships
+      # config/redis.config.php, which -- when REDIS_HOST is set -- declares
+      # memcache.distributed, memcache.locking AND the connection block.
+      # Verified against the copy in this instance's persisted config volume.
+      #
+      # Do NOT also `occ config:system:set` those keys. occ writes config.php,
+      # but Nextcloud merges every *.config.php drop-in AFTER it, so the
+      # drop-in wins -- config.php would read as authoritative while being
+      # silently overridden. memcache.local stays APCu (apcu.config.php).
+      #
+      # REDIS_HOST_PASSWORD_FILE is NOT usable here: the drop-in in this
+      # volume predates that feature and reads only REDIS_HOST_PASSWORD.
+      REDIS_HOST: skudak-cloud-redis
+      REDIS_HOST_PORT: "6379"
+      REDIS_HOST_PASSWORD: "{{ cloud_skudak_redis_pass }}"
    volumes:
      - "{{ cloud_skudak_path }}/apps:/var/www/html/custom_apps"
      - "{{ cloud_skudak_path }}/data:/var/www/html/data"
      - "{{ cloud_skudak_path }}/config:/var/www/html/config"
-      - "{{ cloud_skudak_path }}/scripts/libresign-setup.sh:/docker-entrypoint-hooks.d/before-starting/libresign-setup.sh:ro"
    ports:
      - "8090:80"

@@ -96,20 +205,208 @@
  vars:
    container_name: skudak-cloud

-# Install poppler-utils for pdfsig/pdfinfo (LibreSign handles java/pdftk/jsignpdf via occ)
-# This needs to be reinstalled on each container recreation
- name: install poppler-utils in skudak-cloud
+# ---------------------------------------------------------------------------
+# LibreSign (e-signature for Skudak agreements)
+#
+# poppler-utils supplies pdfsig/pdfinfo; ghostscript is used for PDF
+# normalisation. Both land in /usr, which is NOT a persisted volume, so they
+# must be reinstalled after every container recreation. Java, PDFtk and
+# jSignPdf are different -- LibreSign installs those under
+# data/appdata_*/libresign, which IS persisted, so they survive.
+- name: install libresign runtime dependencies in skudak-cloud
  become: true
  become_user: "{{ podman_user }}"
  ansible.builtin.command:
    cmd: >
      podman exec -u 0 skudak-cloud
      sh -c "apt-get update && apt-get install -y --no-install-recommends
-      poppler-utils && rm -rf /var/lib/apt/lists/*"
-  register: poppler_install
-  changed_when: "'is already the newest version' not in poppler_install.stdout"
+      poppler-utils ghostscript && rm -rf /var/lib/apt/lists/*"
+  register: libresign_deps
+  changed_when: "'is already the newest version' not in libresign_deps.stdout"
+
+# When the container is recreated, the entrypoint re-extracts Nextcloud into
+# the /var/www/html volume before Apache starts. Every occ call below races
+# that: it fails with "Failed opening required .../lib/versioncheck.php" until
+# the extraction completes. Poll until occ answers rather than sleeping a
+# fixed interval, which would be both slower and still unreliable.
+- name: wait for nextcloud to be ready in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ status --output=json
+  register: skudak_occ_ready
+  until: skudak_occ_ready.rc == 0 and 'installed' in skudak_occ_ready.stdout
+  retries: 30
+  delay: 5
+  changed_when: false
+
+# A disabled app deregisters every `occ libresign:*` command, which makes the
+# app look uninstalled rather than switched off. Found disabled on 2026-07-31.
+- name: ensure libresign app is enabled in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ app:enable libresign
+  register: libresign_enable
+  changed_when: "'already enabled' not in libresign_enable.stdout"
+
+# Ensure-installed: LibreSign no-ops when the binaries are already present
+# under data/appdata_*/libresign (a persisted volume). It prints "Finished with
+# success." either way and gives no signal distinguishing a fresh download from
+# a no-op, so this never reports changed rather than reporting it every run.
+- name: install libresign java/pdftk/jsignpdf binaries in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ libresign:install --java --pdftk --jsignpdf
+  register: libresign_install
+  changed_when: false
+  failed_when: "'Finished with success' not in libresign_install.stdout"
+
+- name: check whether libresign root certificate is configured
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ libresign:configure:check --certificate
+  register: libresign_cert_check
+  changed_when: false
  failed_when: false

+# Guarded deliberately. Running this unconditionally would mint a new root CA
+# on every deploy and invalidate every certificate already issued to a signer,
+# breaking the trust chain on documents that were already signed.
+#
+# Do NOT add --ou here: LibreSign appends its own `libresign-ca-id:...` entry
+# to the OU field, and the combined value overruns the 64-character ASN.1
+# limit for organizationalUnitName, failing with "string too long".
+- name: generate libresign root certificate for skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ libresign:configure:openssl
+    --cn="{{ libresign_skudak_cert_cn }}"
+    -o "{{ libresign_skudak_cert_o }}"
+    -c "{{ libresign_skudak_cert_c }}"
+    -s "{{ libresign_skudak_cert_st }}"
+    -l "{{ libresign_skudak_cert_l }}"
+  when: "'error' in libresign_cert_check.stdout"
+  changed_when: true
+
+# LibreSign defaults to requiring every signer to upload an identification
+# document, which then needs approval by a member of `approval_group` before
+# the sign action unlocks. For three partners signing their own partnership
+# instruments that is pure friction -- the emailed invitation is the identity
+# check. Without this, signers see "Upload file" and no way to sign.
+- name: relax libresign identification-document gate in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:app:set libresign identification_documents --value=0
+  register: libresign_ident
+  changed_when: "'is now set to' in libresign_ident.stdout"
+
+# SIGNAME_AND_DESCRIPTION (the LibreSign default) typesets the signer's NAME as
+# text and offers no drawing surface at all. GRAPHIC is the mode that asks for
+# an actual signature graphic -- drawn, uploaded or typed -- and stamps ONLY
+# that mark, with no description block.
+#
+# Deliberately GRAPHIC_ONLY rather than the LibreSign default of
+# GRAPHIC_AND_DESCRIPTION. In the latter,
+# SignatureTextService::getSignatureWidth() returns `$current / 2` whenever a
+# text template is set, splitting the stamp into a graphic half and a text
+# half. Our documents already typeset the signer's printed name and the date
+# either side of the signature rule (\signatureblock in skudak-contract.cls),
+# so LibreSign's own name/date block is both redundant and prone to colliding
+# with the drawn mark. GRAPHIC_ONLY takes the early return in that method,
+# using the full width to stamp the signature alone.
+#
+# The value MUST be exactly 'GRAPHIC_ONLY' -- see
+# SignerElementsService::RENDER_MODE_GRAPHIC_ONLY (line 25). 'GRAPHIC' is NOT
+# a valid constant; setting it writes a value the admin UI cannot match to any
+# radio button, silently reverting the effective behaviour to the default.
+- name: use signature-only stamp in skudak-cloud libresign
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:app:set libresign signature_render_mode --value=GRAPHIC_ONLY
+  register: libresign_render
+  changed_when: "'is now set to' in libresign_render.stdout"
+
+# LibreSign stamps a validation footer onto EVERY page of a signed PDF
+# (FooterHandler::getFooter(), default on). The QR block within it is a large
+# square that lands in the same band as our own document footer -- the rule,
+# "Page N of M" and the Skudak mark set by skudak-contract.cls -- and overlaps
+# it.
+#
+# The QR is dropped; the "Digitally signed by ... Validate in <url>" TEXT is
+# deliberately KEPT. That line is how a recipient independently verifies who
+# signed, when, and under which certificate, which matters for instruments that
+# may have to stand up in diligence. Only the redundant graphic goes -- the URL
+# it encodes remains printed beside it.
+#
+# Must be written with --type=boolean: FooterHandler reads it via
+# getValueBool() (line 158), and the typed appconfig API does not coerce a
+# string "0" to false.
+# Lets an email that belongs to an existing Nextcloud account be added as a
+# LibreSign signer. Arbitrary external addresses already worked; ONLY
+# account-owned ones failed, with a bare "No signers." and nothing logged.
+#
+# Root cause is in Nextcloud core, not LibreSign --
+# lib/private/Collaboration/Collaborators/MailPlugin.php:128-164. On an exact
+# email match against the local system address book, with
+# shareeEnumerationFullMatch on (its default), the plugin adds a TYPE_USER
+# result and returns false. LibreSign registers that plugin as
+# MailByMailPlugin with shareType = TYPE_EMAIL, so the TYPE_USER branch is
+# skipped, nothing is added, and the early return still fires -- never
+# reaching line 243 where the free-form email result is synthesised.
+#
+# Safe here: shareapi_allow_share_dialog_user_enumeration is already at its
+# default 'yes', so users are discoverable by partial search regardless. This
+# changes how exact matches are handled, not who can be found.
+#
+# DO NOT set shareapi_restrict_user_enumeration_full_match_email to 'no'. That
+# hits an early bail at MailPlugin.php:67-69 and disables email signer search
+# ENTIRELY, including the arbitrary-address case that works today. The verify
+# script asserts it has not been set that way.
+- name: allow account-owned emails as libresign signers
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:app:set core
+    shareapi_restrict_user_enumeration_full_match --value=no
+  register: skudak_enum_fullmatch
+  changed_when: "'is now set to' in skudak_enum_fullmatch.stdout"
+
+- name: drop libresign validation QR code from signed-PDF footer
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:app:set libresign write_qrcode_on_footer
+    --value=0 --type=boolean
+  register: libresign_qr
+  changed_when: "'is now set to' in libresign_qr.stdout"
+
+# The whole point of this block. Previously every step ended in `|| echo`, so
+# a broken LibreSign deployed clean and stayed broken for six months.
+- name: verify libresign configuration in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ libresign:configure:check
+  register: libresign_verify
+  changed_when: false
+  failed_when: libresign_verify.stdout is search('\berror\b')
+
 - name: disable nextcloud signup link in config
  become: true
  ansible.builtin.lineinfile:
@@ -155,6 +452,111 @@
  changed_when: "'System config value log_rotate_size' in skudak_log_rotate.stdout"
  failed_when: false

+# ---------------------------------------------------------------------------
+# Skudak mail branding
+#
+# custom_apps IS a persisted bind mount, so the app survives container
+# recreation; only enabling it and the config values need reasserting.
+- name: deploy skudakmail email-template app to skudak-cloud
+  become: true
+  ansible.builtin.copy:
+    src: skudakmail/
+    dest: "{{ cloud_skudak_path }}/apps/skudakmail/"
+    owner: "{{ podman_subuid.stdout }}"
+    group: "{{ podman_subuid.stdout }}"
+    mode: 0644
+    directory_mode: 0755
+  notify: restorecon podman
+
+- name: unshare chown skudakmail app
+  become: true
+  become_user: "{{ podman_user }}"
+  changed_when: false
+  ansible.builtin.command: >
+    podman unshare chown -R 33:33 {{ cloud_skudak_path }}/apps/skudakmail
+
+- name: enable skudakmail app in skudak-cloud
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud php occ app:enable skudakmail
+  register: skudakmail_enable
+  changed_when: "'already enabled' not in skudakmail_enable.stdout"
+
+# Supported extension point -- Mailer::createEMailTemplate() checks this and
+# instantiates the named class if it extends EMailTemplate. Not a core patch.
+- name: point nextcloud at the skudak email template
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:system:set mail_template_class
+    --value={{ "OCA\\Skudakmail\\Mail\\SkudakEMailTemplate" }}
+  register: skudak_mail_class
+  changed_when: "'set to' in skudak_mail_class.stdout"
+
+# Email asset URLs and every link LibreSign puts in a signature invitation are
+# built from overwrite.cli.url when sending from a background job. It pointed
+# at the pre-rename cloud.skudakrennsport.com, so invitations carried the old
+# domain and the logo <img> resolved against it.
+- name: set skudak-cloud canonical cli url
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud
+    php occ config:system:set overwrite.cli.url
+    --value=https://{{ cloud_skudak_server_name_new }}
+  register: skudak_cli_url
+  changed_when: "'set to' in skudak_cli_url.stdout"
+
+# Theming that the email template reads. The logo MUST be a wide, tightly
+# cropped image: Mailer clamps to MAX_LOGO_SIZE=105 preserving aspect, so a
+# SQUARE logo renders as a 105x105 block in a coloured band -- which is
+# exactly how an 8334x8334 upload turned the header into a giant blue blob.
+- name: set skudak-cloud theming
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.command: >
+    podman exec -u www-data skudak-cloud php occ theming:config {{ item.k }} "{{ item.v }}"
+  loop:
+    - {k: name, v: "Skudak"}
+    - {k: slogan, v: "Aftermarket vintage car parts and restoration"}
+    - {k: url, v: "https://skudak.com"}
+    - {k: primary_color, v: "{{ theming_skudak_primary }}"}
+    - {k: background_color, v: "{{ theming_skudak_primary }}"}
+  register: skudak_theming
+  changed_when: "'Updated' in skudak_theming.stdout"
+  loop_control:
+    label: "{{ item.k }}"
+
+# Branding rides on OC\Mail\EMailTemplate, which is Nextcloud's PRIVATE
+# namespace with no API stability guarantee. A Nextcloud major upgrade disables
+# the app (info.xml pins max-version), Mailer falls back to the stock template,
+# and mail keeps sending -- unbranded and silent. This turns that silence into
+# a failed play. Renders through Message::useTemplate(), the real path, and
+# also re-asserts the LibreSign signing settings.
+- name: template skudakmail verification script
+  become: true
+  ansible.builtin.template:
+    src: nextcloud/skudakmail-verify.php.j2
+    dest: "{{ cloud_skudak_path }}/scripts/skudakmail-verify.php"
+    owner: "{{ podman_subuid.stdout }}"
+    group: "{{ podman_subuid.stdout }}"
+    mode: 0644
+  notify: restorecon podman
+
+- name: verify skudak mail branding is live
+  become: true
+  become_user: "{{ podman_user }}"
+  ansible.builtin.shell: >
+    set -o pipefail;
+    podman exec -i -u www-data skudak-cloud php
+    < {{ cloud_skudak_path }}/scripts/skudakmail-verify.php
+  args:
+    executable: /bin/bash
+  register: skudakmail_verify
+  changed_when: false
+
 - include_tasks: containers/cloud-cron.yml
  vars:
    cron_name: skudak-cloud
@@ -183,10 +585,18 @@
 # cannot repeat. The personal task's `/skudakcloud/**` exclude is PERMANENT --
 # it is what keeps business data out of personal storage, not a stopgap.
 #
-# Still outstanding: the data itself lives on personal TrueNAS hardware. To
-# finish separating, add an S3 stage to cloud-backup.sh.j2 guarded by a
-# `backup_s3_*` var so only this instance opts in -- awscli2 is already
-# installed on the host -- and then drop the TrueNAS rsync below.
+# A direct host-to-iDrive S3 stage was built here and then REMOVED on
+# 2026-07-31. It would have written the same data into the same `backup-all`
+# bucket that the TrueNAS cloud-sync task above already fills -- duplicate
+# storage, two writers to one prefix, for no additional coverage. Offsite to
+# business-owned storage was already solved by that cloud-sync task; the
+# earlier note in this file proposed adding S3 *and then dropping the rsync*,
+# i.e. replacement, and building both was a misreading of it.
+#
+# If offsite is ever moved onto this host, it must REPLACE the rsync below,
+# not run beside it. The open question to settle first is whether the
+# TrueNAS -> iDrive leg is independently verifiable; nobody has confirmed that
+# task's run history end to end, and keeping this chain means trusting it.
 - include_tasks: containers/cloud-backup.yml
  vars:
    backup_name: skudak-cloud
@@ -5,6 +5,11 @@
 - import_tasks: podman/podman-prune.yml
  tags: podman-prune

+# Heals the TrueNAS CIFS mounts and bounces immich when they return.
+# Must be deployable independently of the photos containers.
+- import_tasks: podman/cifs-watchdog.yml
+  tags: cifs-watchdog, photos
+
 # WEB SERVER: Caddy is the default and only web server
 # nginx has been completely replaced and removed

@@ -70,6 +75,7 @@
 - import_tasks: containers/skudak/cloud.yml
  vars:
    db_image: docker.io/library/mariadb:10.6
+    redis_image: docker.io/redis:8.2-alpine
    image: docker.io/library/nextcloud:34.0.2-apache
  tags: skudak, skudak-cloud

@@ -0,0 +1,36 @@
+---
+- name: template cifs watchdog script
+  become: true
+  ansible.builtin.template:
+    src: cifs-watchdog.sh.j2
+    dest: /usr/local/bin/cifs-watchdog.sh
+    owner: root
+    group: root
+    mode: 0755
+    setype: bin_t
+
+- name: template cifs watchdog systemd service
+  become: true
+  ansible.builtin.template:
+    src: cifs-watchdog.service.j2
+    dest: /etc/systemd/system/cifs-watchdog.service
+    owner: root
+    group: root
+    mode: 0644
+
+- name: template cifs watchdog systemd timer
+  become: true
+  ansible.builtin.template:
+    src: cifs-watchdog.timer.j2
+    dest: /etc/systemd/system/cifs-watchdog.timer
+    owner: root
+    group: root
+    mode: 0644
+
+- name: enable and start cifs watchdog timer
+  become: true
+  ansible.builtin.systemd:
+    name: cifs-watchdog.timer
+    enabled: true
+    state: started
+    daemon_reload: true
@@ -0,0 +1,8 @@
+# {{ ansible_managed }}
+# CIFS credentials for the TrueNAS shares consumed by immich.
+#
+# This file exists so the password stops living in /etc/fstab, which is
+# world-readable (0644) by design -- every local user could read the SMB
+# credential straight out of it. Mounted via credentials= instead.
+username={{ cifs_username }}
+password={{ cifs_password }}
@@ -0,0 +1,13 @@
+[Unit]
+Description=Verify TrueNAS CIFS mounts and heal immich after recovery
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/cifs-watchdog.sh
+# Type=oneshot disables the start timeout by default. Every external call in
+# the script is already wrapped in `timeout`, but bound the unit too so a
+# wedged run cannot block every subsequent trigger.
+TimeoutStartSec={{ cifs_watchdog_timeout | default('5m') }}
+Nice=10
@@ -0,0 +1,97 @@
+#!/bin/bash
+# {{ ansible_managed }}
+# Keep the TrueNAS CIFS mounts healthy and heal immich when they come back.
+#
+# Why this exists at all: the immich containers are systemd USER units under
+# "{{ podman_user }}", while the mounts are SYSTEM units. A user unit cannot
+# declare RequiresMountsFor= against a system mount, so there is no native way
+# to say "restart immich when this mount returns". This bridges the two scopes.
+#
+# The failure it addresses: TrueNAS was power-cycled, the .mount units failed,
+# and systemd never retried -- mount units are not restarted on failure. SMB
+# came back, nothing remounted, and immich-server kept serving an empty library
+# while its database still listed 14k assets.
+#
+# Deliberately NOT `set -e`: one unhealthy mount must not stop the others from
+# being checked or healed. Same reasoning as nextcloud-backup-alert.sh.j2.
+set -uo pipefail
+
+TAG=cifs-watchdog
+STATE_DIR=/var/lib/cifs-watchdog
+
+log()  { logger -t "$TAG" -p daemon.info -- "$*"; echo "$TAG: $*"; }
+warn() { logger -t "$TAG" -p daemon.err  -- "$*"; echo "$TAG: $*" >&2; }
+
+install -d -m 0755 "$STATE_DIR"
+
+healed=0
+
+for mp in {{ cifs_watchdog_mounts | map('quote') | join(' ') }}; do
+  key="$STATE_DIR/$(systemd-escape -p "$mp")"
+  prev="unknown"
+  [ -f "$key" ] && prev="$(cat "$key" 2>/dev/null)"
+
+  # `mountpoint -q` alone is NOT sufficient: a CIFS mount whose server vanished
+  # stays "mounted" while every read returns EIO. The directory listing is the
+  # real health check. timeout guards against a hang despite `soft`.
+  if timeout 15 ls -1 "$mp" >/dev/null 2>&1 && mountpoint -q "$mp" 2>/dev/null; then
+    state=healthy
+  else
+    state=unhealthy
+  fi
+
+  recovered=0
+  if [ "$state" = unhealthy ]; then
+    warn "mount=$mp status=unhealthy action=recovering"
+    unit="$(systemd-escape -p --suffix=mount "$mp")"
+    automount="$(systemd-escape -p --suffix=automount "$mp")"
+    # A unit left in `failed` refuses to start again until it is reset.
+    systemctl reset-failed "$unit" "$automount" 2>/dev/null
+    # Start the .mount unit DIRECTLY -- do not try to trigger the automount by
+    # listing the path. An unmounted mount point is still an ordinary empty
+    # directory, so `ls` succeeds and tells us nothing; relying on it silently
+    # skipped recovery entirely in testing.
+    timeout 40 systemctl start "$unit" 2>/dev/null
+    # Re-arm the on-access trigger too, so a drop between watchdog ticks is
+    # repaired by the next process that touches the path rather than waiting.
+    systemctl start "$automount" 2>/dev/null
+    if timeout 15 ls -1 "$mp" >/dev/null 2>&1 && mountpoint -q "$mp" 2>/dev/null; then
+      state=healthy
+      recovered=1
+      log "mount=$mp status=recovered"
+    else
+      warn "mount=$mp status=still-unhealthy"
+    fi
+  fi
+
+  # Restart when the mount became healthy after being down -- either across
+  # ticks (prev=unhealthy) or WITHIN this run (recovered=1). The second case is
+  # not redundant: a drop that is detected and repaired by the same invocation
+  # leaves prev=healthy, so checking only the stored state silently skips the
+  # restart while the container still holds its stale, empty view.
+  #
+  # Still gated on actually reaching healthy, so a run that fails to recover
+  # does not bounce immich on every tick while the NAS is down.
+  if [ "$state" = healthy ] && { [ "$prev" = unhealthy ] || [ "$recovered" -eq 1 ]; }; then
+    healed=1
+  fi
+
+  echo "$state" > "$key"
+done
+
+if [ "$healed" -eq 1 ]; then
+  # ONLY immich-server: it is the sole consumer of both CIFS paths. Postgres,
+  # redis and machine-learning use local volumes and must not be bounced.
+  #
+  # Rootless podman under "{{ podman_user }}": -H for HOME, the `cd;` preamble
+  # is required (see CLAUDE.md), and XDG_RUNTIME_DIR for systemctl --user.
+  if sudo -H -u {{ podman_user }} bash -c \
+       'cd; export XDG_RUNTIME_DIR=/run/user/$(id -u)
+        systemctl --user restart {{ cifs_watchdog_restart_unit | default("immich-server.service") }}' 2>/dev/null; then
+    log "status=healed action=restarted unit={{ cifs_watchdog_restart_unit | default('immich-server.service') }}"
+  else
+    warn "status=healed action=restart-FAILED unit={{ cifs_watchdog_restart_unit | default('immich-server.service') }}"
+  fi
+fi
+
+exit 0
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic TrueNAS CIFS mount health check
+
+[Timer]
+OnBootSec={{ cifs_watchdog_onbootsec | default('2m') }}
+OnCalendar={{ cifs_watchdog_oncalendar }}
+RandomizedDelaySec=30s
+# Not Persistent: this is a liveness check, so a run missed while the host was
+# off carries no meaning -- the next tick reflects current reality.
+Persistent=false
+
+[Install]
+WantedBy=timers.target
@@ -171,5 +171,4 @@ log "syncing db dumps"
 rsync -az --timeout=600 --delete --mkpath {{ backup_rsync_extra_args | default('') }} \
  -e "$SSH" "$STAGE/db/" "$DEST:{{ remote_path }}/_backup/db/"
 {% endif %}
-
 log "status=ok"
@@ -1,22 +0,0 @@
-#!/bin/bash
-# LibreSign dependency setup for Skudak Nextcloud
-# Runs on container start via /docker-entrypoint-hooks.d/before-starting/
-# Note: This runs as www-data, not root. poppler-utils is installed
-# separately via Ansible using podman exec -u 0.
-
-echo "=== LibreSign Setup: Installing dependencies ==="
-
-# Install LibreSign-managed Java (required for PDFtk and jSignPdf)
-# This downloads a specific Java version that LibreSign validates
-echo "Installing Java..."
-php /var/www/html/occ libresign:install --java || echo "Java install skipped or failed"
-
-# Install PDFtk (requires Java)
-echo "Installing PDFtk..."
-php /var/www/html/occ libresign:install --pdftk || echo "PDFtk install skipped or failed"
-
-# Install jSignPdf (requires Java)
-echo "Installing jSignPdf..."
-php /var/www/html/occ libresign:install --jsignpdf || echo "jSignPdf install skipped or failed"
-
-echo "=== LibreSign Setup: Complete ==="
@@ -0,0 +1,40 @@
+# {{ ansible_managed }}
+#
+# Redis for skudak-cloud: Nextcloud distributed cache + transactional file
+# locking. Reachable only by container name on the `shared` podman network --
+# no host port is published.
+#
+# The password lives HERE rather than on the command line as
+# `redis-server --requirepass <pass>`. That is the existing house idiom (see
+# the deleted container-nosql.yml in git history), but it leaks the secret into
+# `podman inspect`, into the generated systemd unit under
+# ~/.config/systemd/user/, and into `ps` for every user on the host. A 0640
+# config file mounted read-only keeps it out of all three.
+requirepass {{ cloud_skudak_redis_pass }}
+
+# Bind to all interfaces WITHIN the container's network namespace. The
+# container publishes no port, so this is reachable only from the `shared`
+# podman network -- not from the host and not from the LAN.
+bind 0.0.0.0
+port 6379
+protected-mode yes
+
+# NO maxmemory / eviction policy, deliberately.
+#
+# Nextcloud puts BOTH the distributed cache and the transactional file locks in
+# this instance. Cache entries are safely evictable; LOCKS ARE NOT. An
+# `allkeys-lru` policy under memory pressure can evict a lock that a live
+# request still believes it holds, which permits concurrent writers to the same
+# file -- silent corruption rather than a visible error. With no maxmemory,
+# Redis never evicts. The host has ~14 GiB free of 31 GiB and this instance
+# holds a few hundred keys, so a cap buys nothing.
+#
+# If a cap is ever genuinely needed, use `maxmemory-policy noeviction` so Redis
+# returns an error instead of silently discarding a lock.
+
+# No persistence. Locks are ephemeral and TTL-bounded, and the cache is
+# rebuildable -- there is nothing here worth surviving a restart. Persisting
+# would be actively worse: a restored RDB could reinstate locks whose owning
+# request died, blocking files until the TTL expired.
+save ""
+appendonly no
@@ -0,0 +1,186 @@
+<?php
+/**
+ * {{ ansible_managed }}
+ *
+ * Post-deploy assertion that Skudak mail branding is actually live.
+ *
+ * WHY THIS EXISTS: SkudakEMailTemplate extends OC\Mail\EMailTemplate, which is
+ * Nextcloud's PRIVATE namespace -- no API stability guarantee. Two things can
+ * silently switch the branding off:
+ *
+ *   1. A Nextcloud major upgrade. appinfo/info.xml pins max-version, so the app
+ *      is auto-disabled as incompatible; Mailer::createEMailTemplate() then
+ *      fails its class_exists() check and falls back to the stock template.
+ *      Mail still sends -- unbranded. That is the right failure mode, but it is
+ *      invisible without this check.
+ *   2. An upstream change to the private base class breaking an override.
+ *
+ * Renders through Message::useTemplate() -- the REAL path -- rather than
+ * calling renderHtml() directly. That distinction is not academic: renderText()
+ * runs first and flips the parent's footerAdded flag, and a renderHtml()-only
+ * test once passed green while live mail shipped with no footer at all.
+ *
+ * Exits non-zero with a diagnostic on any failure, so the Ansible task fails
+ * the play rather than reporting a clean deploy over broken branding.
+ */
+
+require_once '/var/www/html/lib/base.php';
+
+$mailer = \OC::$server->get(\OCP\Mail\IMailer::class);
+$dispatcher = \OC::$server->get(\OCP\EventDispatcher\IEventDispatcher::class);
+
+// Mirrors MailService::notifyUnsignedUser() (custom_apps/libresign/lib/Service/MailService.php:85-116).
+$template = $mailer->createEMailTemplate('settings.TestEmail');
+$template->setSubject('LibreSign: There is a file for you to sign');
+$template->addHeader();
+$template->addHeading('File to sign', false);
+$template->addBodyText('There is a document for you to sign. Access the link below:');
+$template->addBodyButton('Sign »verify.pdf«', 'https://{{ cloud_skudak_server_name_new }}/verify');
+
+$message = $mailer->createMessage();
+$message->setTo(['verify@example.invalid' => 'Verify']);
+$message->useTemplate($template);
+
+// What Mailer::send() does at lib/private/Mail/Mailer.php:186. Nothing is sent.
+$dispatcher->dispatchTyped(new \OCP\Mail\Events\BeforeMessageSent($message));
+
+$html = $message->getSymfonyEmail()->getHtmlBody() ?? '';
+$text = $message->getPlainBody();
+$subject = $message->getSubject();
+
+$inlineNames = [];
+foreach ($message->getSymfonyEmail()->getAttachments() as $part) {
+	$inlineNames[] = (string)$part->getFilename();
+}
+
+$failures = [];
+
+if (!$template instanceof \OCA\Skudakmail\Mail\SkudakEMailTemplate) {
+	$failures[] = 'template class is ' . get_class($template)
+		. ' -- expected SkudakEMailTemplate. Is the skudakmail app enabled, and does '
+		. 'appinfo/info.xml still allow this Nextcloud major?';
+}
+if (!str_starts_with($subject, 'Skudak LLP')) {
+	$failures[] = 'subject not rewritten: ' . $subject;
+}
+if (!str_contains($html, 'official document-signing request')) {
+	$failures[] = 'HTML footer missing (renderText/renderHtml ordering regression?)';
+}
+if (!str_contains($text, 'official document-signing request')) {
+	$failures[] = 'plain-text footer missing';
+}
+if (!str_contains($html, 'skudak.com/privacy') || !str_contains($html, 'skudak.com/terms')) {
+	$failures[] = 'privacy/terms links missing from footer';
+}
+if (preg_match('/[»«]/u', $html)) {
+	$failures[] = 'German guillemets survived into the body';
+}
+if (!str_contains($html, 'Review document')) {
+	$failures[] = 'button label not normalised to "Review document"';
+}
+if (!str_contains($html, 'cid:skudak-wordmark.png')) {
+	$failures[] = 'logo is not a cid: reference -- BeforeMessageSent listener did not fire';
+}
+if (!in_array('skudak-wordmark.png', $inlineNames, true)) {
+	$failures[] = 'inline logo MIME part absent (found: ' . (implode(', ', $inlineNames) ?: 'none') . ')';
+}
+
+// ---------------------------------------------------------------------------
+// LibreSign signing settings. These live in oc_appconfig (the database), not on
+// disk, so they survive container recreation -- but they are re-assertable and
+// a stray click in the admin UI can change them silently. GRAPHIC in particular
+// matters: any other mode makes SignatureTextService::getSignatureWidth()
+// return $current / 2 and stamp a name/date block that duplicates -- and
+// collides with -- the one our documents already typeset.
+$appConfig = \OC::$server->get(\OCP\IAppConfig::class);
+
+// Must be exactly GRAPHIC_ONLY -- SignerElementsService::RENDER_MODE_GRAPHIC_ONLY.
+// The valid set is DESCRIPTION_ONLY / SIGNAME_AND_DESCRIPTION /
+// GRAPHIC_AND_DESCRIPTION / GRAPHIC_ONLY. Anything outside it (a bare 'GRAPHIC',
+// say) is accepted by occ but matches no radio in the admin UI and falls
+// through to default behaviour, so this asserts membership, not just non-empty.
+$renderMode = $appConfig->getValueString('libresign', 'signature_render_mode', '');
+if ($renderMode !== 'GRAPHIC_ONLY') {
+	$failures[] = 'libresign signature_render_mode is "' . $renderMode
+		. '" -- expected GRAPHIC_ONLY (signature only). Any other mode halves the '
+		. 'stamp width and overlays a duplicate name/date block.';
+}
+
+// Read with getValueBool, exactly as FooterHandler:158 does -- asserting the
+// string form would pass on a value the app itself reads as true.
+if ($appConfig->getValueBool('libresign', 'write_qrcode_on_footer', true) !== false) {
+	$failures[] = 'libresign write_qrcode_on_footer is not false -- the validation '
+		. 'QR block will be stamped on every page and overlaps the document footer '
+		. 'set by skudak-contract.cls. (Was it written without --type=boolean?)';
+}
+
+// Signer search for account-owned emails. Both keys are asserted because the
+// two failure modes are opposite and the second is the more dangerous:
+//   full_match = yes        -> account-owned emails silently unselectable
+//   full_match_email = no   -> email signer search disabled ENTIRELY
+// Defaults are 'yes' for both (MailPlugin.php:50-55), so an unset
+// full_match_email is correct and only an explicit 'no' is a problem.
+if ($appConfig->getValueString('core', 'shareapi_restrict_user_enumeration_full_match', 'yes') !== 'no') {
+	$failures[] = 'core shareapi_restrict_user_enumeration_full_match is not "no" -- '
+		. 'emails belonging to an existing Nextcloud account cannot be added as '
+		. 'LibreSign signers (MailPlugin.php:163 aborts the search).';
+}
+if ($appConfig->getValueString('core', 'shareapi_restrict_user_enumeration_full_match_email', 'yes') === 'no') {
+	$failures[] = 'core shareapi_restrict_user_enumeration_full_match_email is "no" -- '
+		. 'this disables email signer search ENTIRELY (MailPlugin.php:67). It must be '
+		. 'unset or "yes"; it is NOT the knob for the account-owned-email problem.';
+}
+
+$identDocs = $appConfig->getValueString('libresign', 'identification_documents', '');
+if ($identDocs !== '0') {
+	$failures[] = 'libresign identification_documents is "' . $identDocs
+		. '" -- expected 0. A non-zero value gates signing behind an ID upload '
+		. 'plus admin approval, and signers see no way to sign.';
+}
+
+// ---------------------------------------------------------------------------
+// Redis: distributed cache + transactional file locking.
+//
+// These come from the image's config/redis.config.php drop-in, which only
+// activates when REDIS_HOST is set on the container. If the env var is lost
+// (a container recreated from a stale spec, say), Nextcloud silently reverts
+// to DBLockingProvider and every file lock goes back to being a MariaDB write
+// -- functional, but the stalls come back with no error anywhere.
+$sysConfig = \OC::$server->get(\OCP\IConfig::class);
+
+foreach (['memcache.locking', 'memcache.distributed'] as $key) {
+	$value = $sysConfig->getSystemValueString($key, '');
+	if ($value !== '\OC\Memcache\Redis') {
+		$failures[] = $key . ' is "' . $value . '" -- expected \\OC\\Memcache\\Redis. '
+			. 'Is REDIS_HOST still set on the skudak-cloud container?';
+	}
+}
+
+// Prove Redis is actually reachable and authenticating, not merely configured.
+// A wrong password leaves the config looking perfect while every cache and
+// lock operation fails at runtime.
+try {
+	$cacheFactory = \OC::$server->get(\OCP\ICacheFactory::class);
+	if (!$cacheFactory->isAvailable()) {
+		$failures[] = 'distributed cache reports unavailable -- redis unreachable or auth failed';
+	} else {
+		$probe = $cacheFactory->createDistributed('skudakmail-verify');
+		$probe->set('probe', 'ok', 30);
+		if ($probe->get('probe') !== 'ok') {
+			$failures[] = 'distributed cache round-trip failed (set/get mismatch)';
+		}
+		$probe->remove('probe');
+	}
+} catch (\Throwable $e) {
+	$failures[] = 'distributed cache threw: ' . $e->getMessage();
+}
+
+if ($failures !== []) {
+	fwrite(STDERR, "skudakmail branding verification FAILED:\n");
+	foreach ($failures as $f) {
+		fwrite(STDERR, "  - $f\n");
+	}
+	exit(1);
+}
+
+echo "skudakmail branding OK (subject: $subject)\n";
Author	SHA1	Message	Date
Bastian de Byl	773a2bbc9c	harden TrueNAS CIFS mounts so immich self-heals TrueNAS was power-cycled, the CIFS mounts failed, and systemd never retried -- mount units are not restarted on failure. SMB came back, nothing remounted, and immich-server served an empty library for days while its database still listed 14,181 assets pointing at /mnt/media/originals. fstab carried no _netdev, no nofail and no automount, so there was no path back without a human. Now: x-systemd.automount any access re-attempts the mount; failure stops being terminal _netdev / nofail ordered after network-online, dead NAS cannot block boot soft I/O errors instead of blocking forever, so the container can be restarted rather than wedging in uninterruptible sleep idle-timeout unmount when unused, clearing stale handles resilienthandles SMB3 rides out brief blips ansible.posix.mount mounts directly and never starts the generated .automount unit, leaving the on-access trigger inactive -- enable it explicitly, or the headline fix silently does nothing. The containers are systemd USER units while the mounts are SYSTEM units, so RequiresMountsFor= is unavailable. cifs-watchdog bridges the scopes: checks health, recovers, and restarts ONLY immich-server (the sole consumer of both paths; postgres/redis/ML use local volumes). Two bugs the umount test caught, both worth knowing: - `ls` cannot test mountedness. An unmounted mount point is an ordinary empty directory, so ls succeeds and recovery was skipped entirely. - A drop repaired within a single run leaves prev=healthy, so keying the restart solely on the stored state skipped it while the container still held its stale view. Also moves the SMB password out of /etc/fstab, which is 0644 and was readable by every local user, into a 0600 credentials file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 14:49:22 -04:00
Bastian de Byl	0939204061	docs(podman): add role README with backup, restore and LibreSign runbook No restore procedure existed anywhere in this repo. The backup pipeline is well commented but nothing described how to get data back, and the README the backup script already referenced was missing. Covers the shared backup engine and its stage ordering, a restore procedure for both MariaDB and Postgres with the rootless-podman command form, data-tree restore including the uid 33 chown, and the maintenance-mode/files:scan reconcile. Two things worth stating plainly: - Untested backups are not a control. No rehearsal has been recorded. - Do not blindly re-run libresign:configure:openssl on a restored instance -- it mints a new root CA and invalidates the trust chain on every document already signed under the old one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 15:55:04 -04:00
Bastian de Byl	fec7d62acb	feat(skudak-cloud): repair LibreSign, brand its mail, add Redis LibreSign had been silently broken since it was first deployed in January. Every step of the old before-starting hook ended in `\|\| echo`, so six months of failures logged nothing. LibreSign repair - Root cause was a stale config_path: a valid OpenSSL root CA existed at generation 1, a failed CFSSL attempt left an empty generation 2, and config_path was left pointing at the empty one. Regenerated as "Skudak LLP" (was the pre-rename "Skudak Rennsport LLP"). - Deleted the hook. Java/PDFtk/jSignPdf live under data/appdata_*, a persisted volume, so they only ever needed installing once. Install and verification are now explicit tasks that actually fail. - PHP_MEMORY_LIMIT 1024M -- the 512M image default fails opaquely mid-signature. LC_ALL/LANG so the JVM is not ANSI_X3.4-1968. - signature_render_mode=GRAPHIC_ONLY. Any other mode halves the stamp width and overlays a name/date block that collides with the drawn mark and duplicates what our documents already typeset. The value must be exactly GRAPHIC_ONLY; a bare "GRAPHIC" is accepted by occ, matches no radio in the UI, and silently reverts to default. - write_qrcode_on_footer=false, written with --type=boolean because FooterHandler reads it via getValueBool and the typed appconfig API does not coerce a string "0". The validation URL text is kept. - identification_documents=0 -- the default gates signing behind an ID upload plus admin approval, so signers saw no way to sign. - shareapi_restrict_user_enumeration_full_match=no, so an email owned by an existing account can be added as a signer. Root cause is in core (MailPlugin.php:163), not LibreSign. Do NOT set full_match_email=no -- that disables email signer search entirely. Mail branding (skudakmail app) - Two supported extension points, no core patch and no LibreSign fork: mail_template_class for layout, subjects, button labels and the footer LibreSign never adds; and a BeforeMessageSent listener to embed the wordmark as a cid: part so it survives remote-image blocking. - A third listener adds scoped CSS fixing the signing page being clipped on iOS Safari (100vh -> 100dvh). Patched upstream too. - skudakmail-verify.php.j2 asserts all of the above through the real useTemplate() path and fails the play on drift. Every assertion was proven to fail when deliberately regressed. Redis - memcache.locking was unset, so Nextcloud used DBLockingProvider and every file lock became a MariaDB write -- the contention behind the intermittent multi-second stalls. Verified after: db locks static, redis keys growing. - requirepass lives in a mounted 0640 conf, not --requirepass, which would leak it into podman inspect, the systemd unit and ps. The file is chowned to uid 999 because redis-server does not run as root and the :ro mount stops the image fixing it itself. - No maxmemory: cache is evictable, locks are NOT, and evicting a held lock permits concurrent writers to one file. No persistence either -- a restored RDB could reinstate locks whose owner is long dead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 15:54:52 -04:00
Bastian de Byl	c184099b2d	refactor(backup): remove duplicate direct-to-S3 stage A host-to-iDrive S3 stage was added here and is now removed. It would have written the same data into the same `backup-all` bucket that the TrueNAS cloud-sync task already fills -- duplicate storage, two writers to one prefix, for no additional coverage. Offsite to business-owned storage was already solved: the rsync feeds /mnt/glacier/skudakcloud and TrueNAS cloud-syncs that to Skudak's own iDrive e2 account. The earlier note in skudak/cloud.yml proposed adding S3 and then dropping the rsync -- replacement, not addition -- and building both was a misreading of it. If offsite is ever moved onto this host it must REPLACE the rsync, not run beside it. Settle first whether the TrueNAS -> iDrive leg is independently verifiable; keeping this chain means trusting it. Also removes the now-orphaned /etc/backup_s3 credential file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 15:54:27 -04:00