docs(podman): add role README with backup, restore and LibreSign runbook

No restore procedure existed anywhere in this repo. The backup pipeline
is well commented but nothing described how to get data back, and the
README the backup script already referenced was missing.

Covers the shared backup engine and its stage ordering, a restore
procedure for both MariaDB and Postgres with the rootless-podman command
form, data-tree restore including the uid 33 chown, and the
maintenance-mode/files:scan reconcile.

Two things worth stating plainly:
- Untested backups are not a control. No rehearsal has been recorded.
- Do not blindly re-run libresign:configure:openssl on a restored
  instance -- it mints a new root CA and invalidates the trust chain on
  every document already signed under the old one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/podman/README.md

@@ -0,0 +1,143 @@
+# podman role
+
+Container orchestration for the home server. Containers are defined under
+`tasks/containers/{base,home,skudak,debyltech}/` and wired from
+`tasks/main.yml`, which is where every image tag is pinned.
+
+## Backups
+
+Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one
+backup engine: `tasks/containers/cloud-backup.yml` plus
+`templates/nextcloud/cloud-backup.sh.j2`. Each instance includes it with its own
+vars, producing `/usr/local/bin/<name>-backup.sh` and a systemd timer.
+
+Stages, in order — the ordering is deliberate, see the comments in the template:
+
+1. **Database dump** inside the container (mariadb / mysql / postgres branches),
+   gzipped to `/var/backups/nextcloud/<name>/db/`. Promoted over yesterday's dump
+   only after passing `gzip -t` **and** a completion-trailer grep.
+2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used.
+3. **rsync** of the data tree, config, and db dumps to TrueNAS.
+
+Failures raise `status=failed` on the `nextcloud-backup` syslog tag, which an
+external Graylog rule matches to send mail. Do not rename that tag: it is shared
+by every instance including Gitea, and renaming it here silently stops alerting
+for all of them.
+
+
+## Restore
+
+**Untested backups are not a control.** Rehearse this into a scratch location
+before you need it, and record the date you last did.
+
+### 1. Database
+
+Dumps live on the host at `/var/backups/nextcloud/<name>/db/<name>-YYYYMMDD.sql.gz`
+and on TrueNAS at `<remote_path>/_backup/db/`. TrueNAS in turn cloud-syncs
+`/mnt/glacier/skudakcloud` to Skudak's own iDrive e2 bucket, so a third copy
+exists there — but restoring from it means going through the TrueNAS console,
+not this host.
+
+Verify the dump before trusting it:
+
+```bash
+gzip -t <name>-YYYYMMDD.sql.gz
+gunzip -c <name>-YYYYMMDD.sql.gz | tail -c 512   # expect the completion trailer
+```
+
+Replay into the running database container. MariaDB/MySQL:
+
+```bash
+sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
+  | podman exec -i <db_container> sh -c \
+    "exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"'
+```
+
+Postgres dumps are taken with `--clean --if-exists --no-owner`, so they replay
+into an existing database:
+
+```bash
+sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
+  | podman exec -i <db_container> sh -c \
+    "exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"'
+```
+
+### 2. Data tree
+
+```bash
+# from TrueNAS
+rsync -az -e "ssh -i /etc/ssh/backup_keys/<name>" \
+  <ssh_user>@truenas.localdomain:<remote_path>/ <volumes>/<instance>/data/
+
+```
+
+Then fix ownership — the containers run as uid 33 inside a rootless userns:
+
+```bash
+sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 <volumes>/<instance>/data'
+```
+
+### 3. Reconcile
+
+```bash
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --on'
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ files:scan --all'
+sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --off'
+```
+
+A DB snapshot slightly **older** than the files degrades to "files the app has
+not indexed yet" and is repaired by `files:scan`. A DB snapshot **newer** than
+the files references blobs that were never backed up, which surfaces as broken
+shares and dead file entries — this is why the dump runs first.
+
+### 4. LibreSign-specific
+
+The signing CA lives in the data tree at
+`data/appdata_*/libresign/pki/<instance>_<n>_openssl/`, so a data-tree restore
+brings it back with everything else. After restoring, confirm it:
+
+```bash
+sudo -H -u podman bash -c 'cd; podman exec -u www-data skudak-cloud php occ libresign:configure:check'
+```
+
+Every check must report `success`. If `openssl-configure` reports an error, the
+`certificate_engine` / `config_path` app config is pointing somewhere without a
+CA — see the guarded generate task in `tasks/containers/skudak/cloud.yml`.
+**Do not** simply re-run `libresign:configure:openssl` on a restored instance
+without understanding why: it mints a *new* root CA and invalidates the trust
+chain on every document already signed under the old one.
+
+## LibreSign
+
+Deployed on `skudak-cloud` only. LibreSign 14.1.0 requires Nextcloud server
+`>=34.0.0,<35.0.0`, which the pinned `nextcloud:34.0.2-apache` satisfies. If the
+Nextcloud tag is bumped to 35, LibreSign must be held or upgraded in step — the
+two instances are pinned independently in `tasks/main.yml`, so `skudak-cloud`
+can lag `cloud` if needed.
+
+Dependency split, which drives what survives a container recreate:
+
+| Component | Location | Survives recreate? |
+|---|---|---|
+| Java (JRE 21), PDFtk, jSignPdf | `data/appdata_*/libresign/` | **Yes** — persisted volume |
+| Root CA / PKI | `data/appdata_*/libresign/pki/` | **Yes** — persisted volume |
+| poppler-utils, ghostscript | `/usr` in the image | **No** — reinstalled by Ansible each run |
+
+Certificate engine is **OpenSSL**, not CFSSL. CFSSL is the more common source of
+LibreSign setup failures and buys nothing at this scale.
+
+### Gotchas
+
+- **Do not pass `--ou`** to `libresign:configure:openssl`. LibreSign appends its
+  own `libresign-ca-id:...` entry to the OU field, and the combined value
+  overruns the 64-character ASN.1 limit for `organizationalUnitName`, failing
+  with `string too long`.
+- **A disabled app has no `occ` commands.** If `occ list | grep libresign`
+  returns nothing, the app is disabled, not missing — `occ app:list` will still
+  show it under `Disabled:`. This is what a Nextcloud major upgrade does to an
+  app it thinks is incompatible.
+- `PHP_MEMORY_LIMIT` must be raised above the 512M image default; signing fails
+  opaquely mid-operation otherwise.
+- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and
+  LibreSign warns that accented characters in signer names will be mangled
+  (LibreSign issue #4872).