diff --git a/ansible/roles/podman/README.md b/ansible/roles/podman/README.md new file mode 100644 index 0000000..dde01d7 --- /dev/null +++ b/ansible/roles/podman/README.md @@ -0,0 +1,143 @@ +# podman role + +Container orchestration for the home server. Containers are defined under +`tasks/containers/{base,home,skudak,debyltech}/` and wired from +`tasks/main.yml`, which is where every image tag is pinned. + +## Backups + +Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one +backup engine: `tasks/containers/cloud-backup.yml` plus +`templates/nextcloud/cloud-backup.sh.j2`. Each instance includes it with its own +vars, producing `/usr/local/bin/-backup.sh` and a systemd timer. + +Stages, in order — the ordering is deliberate, see the comments in the template: + +1. **Database dump** inside the container (mariadb / mysql / postgres branches), + gzipped to `/var/backups/nextcloud//db/`. Promoted over yesterday's dump + only after passing `gzip -t` **and** a completion-trailer grep. +2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used. +3. **rsync** of the data tree, config, and db dumps to TrueNAS. + +Failures raise `status=failed` on the `nextcloud-backup` syslog tag, which an +external Graylog rule matches to send mail. Do not rename that tag: it is shared +by every instance including Gitea, and renaming it here silently stops alerting +for all of them. + + +## Restore + +**Untested backups are not a control.** Rehearse this into a scratch location +before you need it, and record the date you last did. + +### 1. Database + +Dumps live on the host at `/var/backups/nextcloud//db/-YYYYMMDD.sql.gz` +and on TrueNAS at `/_backup/db/`. TrueNAS in turn cloud-syncs +`/mnt/glacier/skudakcloud` to Skudak's own iDrive e2 bucket, so a third copy +exists there — but restoring from it means going through the TrueNAS console, +not this host. + +Verify the dump before trusting it: + +```bash +gzip -t -YYYYMMDD.sql.gz +gunzip -c -YYYYMMDD.sql.gz | tail -c 512 # expect the completion trailer +``` + +Replay into the running database container. MariaDB/MySQL: + +```bash +sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \ + | podman exec -i sh -c \ + "exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"' +``` + +Postgres dumps are taken with `--clean --if-exists --no-owner`, so they replay +into an existing database: + +```bash +sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \ + | podman exec -i sh -c \ + "exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"' +``` + +### 2. Data tree + +```bash +# from TrueNAS +rsync -az -e "ssh -i /etc/ssh/backup_keys/" \ + @truenas.localdomain:/ //data/ + +``` + +Then fix ownership — the containers run as uid 33 inside a rootless userns: + +```bash +sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 //data' +``` + +### 3. Reconcile + +```bash +sudo -H -u podman bash -c 'cd; podman exec -u www-data php occ maintenance:mode --on' +sudo -H -u podman bash -c 'cd; podman exec -u www-data php occ files:scan --all' +sudo -H -u podman bash -c 'cd; podman exec -u www-data php occ maintenance:mode --off' +``` + +A DB snapshot slightly **older** than the files degrades to "files the app has +not indexed yet" and is repaired by `files:scan`. A DB snapshot **newer** than +the files references blobs that were never backed up, which surfaces as broken +shares and dead file entries — this is why the dump runs first. + +### 4. LibreSign-specific + +The signing CA lives in the data tree at +`data/appdata_*/libresign/pki/__openssl/`, so a data-tree restore +brings it back with everything else. After restoring, confirm it: + +```bash +sudo -H -u podman bash -c 'cd; podman exec -u www-data skudak-cloud php occ libresign:configure:check' +``` + +Every check must report `success`. If `openssl-configure` reports an error, the +`certificate_engine` / `config_path` app config is pointing somewhere without a +CA — see the guarded generate task in `tasks/containers/skudak/cloud.yml`. +**Do not** simply re-run `libresign:configure:openssl` on a restored instance +without understanding why: it mints a *new* root CA and invalidates the trust +chain on every document already signed under the old one. + +## LibreSign + +Deployed on `skudak-cloud` only. LibreSign 14.1.0 requires Nextcloud server +`>=34.0.0,<35.0.0`, which the pinned `nextcloud:34.0.2-apache` satisfies. If the +Nextcloud tag is bumped to 35, LibreSign must be held or upgraded in step — the +two instances are pinned independently in `tasks/main.yml`, so `skudak-cloud` +can lag `cloud` if needed. + +Dependency split, which drives what survives a container recreate: + +| Component | Location | Survives recreate? | +|---|---|---| +| Java (JRE 21), PDFtk, jSignPdf | `data/appdata_*/libresign/` | **Yes** — persisted volume | +| Root CA / PKI | `data/appdata_*/libresign/pki/` | **Yes** — persisted volume | +| poppler-utils, ghostscript | `/usr` in the image | **No** — reinstalled by Ansible each run | + +Certificate engine is **OpenSSL**, not CFSSL. CFSSL is the more common source of +LibreSign setup failures and buys nothing at this scale. + +### Gotchas + +- **Do not pass `--ou`** to `libresign:configure:openssl`. LibreSign appends its + own `libresign-ca-id:...` entry to the OU field, and the combined value + overruns the 64-character ASN.1 limit for `organizationalUnitName`, failing + with `string too long`. +- **A disabled app has no `occ` commands.** If `occ list | grep libresign` + returns nothing, the app is disabled, not missing — `occ app:list` will still + show it under `Disabled:`. This is what a Nextcloud major upgrade does to an + app it thinks is incompatible. +- `PHP_MEMORY_LIMIT` must be raised above the 512M image default; signing fails + opaquely mid-operation otherwise. +- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and + LibreSign warns that accented characters in signer names will be mangled + (LibreSign issue #4872).