deploy_home/ansible/roles/podman/README.md

podman role

Container orchestration for the home server. Containers are defined under tasks/containers/{base,home,skudak,debyltech}/ and wired from tasks/main.yml, which is where every image tag is pinned.

Backups

Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one backup engine: tasks/containers/cloud-backup.yml plus templates/nextcloud/cloud-backup.sh.j2. Each instance includes it with its own vars, producing /usr/local/bin/<name>-backup.sh and a systemd timer.

Stages, in order — the ordering is deliberate, see the comments in the template:

1. Database dump inside the container (mariadb / mysql / postgres branches), gzipped to /var/backups/nextcloud/<name>/db/. Promoted over yesterday's dump only after passing gzip -t and a completion-trailer grep.
2. SQLite snapshots (.backup, then pragma integrity_check) where used.
3. rsync of the data tree, config, and db dumps to TrueNAS.

Failures raise status=failed on the nextcloud-backup syslog tag, which an external Graylog rule matches to send mail. Do not rename that tag: it is shared by every instance including Gitea, and renaming it here silently stops alerting for all of them.

Restore

Untested backups are not a control. Rehearse this into a scratch location before you need it, and record the date you last did.

1. Database

Dumps live on the host at /var/backups/nextcloud/<name>/db/<name>-YYYYMMDD.sql.gz and on TrueNAS at <remote_path>/_backup/db/. TrueNAS in turn cloud-syncs /mnt/glacier/skudakcloud to Skudak's own iDrive e2 bucket, so a third copy exists there — but restoring from it means going through the TrueNAS console, not this host.

Verify the dump before trusting it:

gzip -t <name>-YYYYMMDD.sql.gz
gunzip -c <name>-YYYYMMDD.sql.gz | tail -c 512   # expect the completion trailer

Replay into the running database container. MariaDB/MySQL:

sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
  | podman exec -i <db_container> sh -c \
    "exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"'

Postgres dumps are taken with --clean --if-exists --no-owner, so they replay into an existing database:

sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
  | podman exec -i <db_container> sh -c \
    "exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"'

2. Data tree

# from TrueNAS
rsync -az -e "ssh -i /etc/ssh/backup_keys/<name>" \
  <ssh_user>@truenas.localdomain:<remote_path>/ <volumes>/<instance>/data/

Then fix ownership — the containers run as uid 33 inside a rootless userns:

sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 <volumes>/<instance>/data'

3. Reconcile

sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --on'
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ files:scan --all'
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --off'

A DB snapshot slightly older than the files degrades to "files the app has not indexed yet" and is repaired by files:scan. A DB snapshot newer than the files references blobs that were never backed up, which surfaces as broken shares and dead file entries — this is why the dump runs first.

4. LibreSign-specific

The signing CA lives in the data tree at data/appdata_*/libresign/pki/<instance>_<n>_openssl/, so a data-tree restore brings it back with everything else. After restoring, confirm it:

sudo -H -u podman bash -c 'cd; podman exec -u www-data skudak-cloud php occ libresign:configure:check'

Every check must report success. If openssl-configure reports an error, the certificate_engine / config_path app config is pointing somewhere without a CA — see the guarded generate task in tasks/containers/skudak/cloud.yml. Do not simply re-run libresign:configure:openssl on a restored instance without understanding why: it mints a new root CA and invalidates the trust chain on every document already signed under the old one.

LibreSign

Deployed on skudak-cloud only. LibreSign 14.1.0 requires Nextcloud server >=34.0.0,<35.0.0, which the pinned nextcloud:34.0.2-apache satisfies. If the Nextcloud tag is bumped to 35, LibreSign must be held or upgraded in step — the two instances are pinned independently in tasks/main.yml, so skudak-cloud can lag cloud if needed.

Dependency split, which drives what survives a container recreate:

Component	Location	Survives recreate?
Java (JRE 21), PDFtk, jSignPdf	data/appdata_*/libresign/	Yes — persisted volume
Root CA / PKI	data/appdata_*/libresign/pki/	Yes — persisted volume
poppler-utils, ghostscript	/usr in the image	No — reinstalled by Ansible each run

Certificate engine is OpenSSL, not CFSSL. CFSSL is the more common source of LibreSign setup failures and buys nothing at this scale.

Gotchas

• Do not pass --ou to libresign:configure:openssl. LibreSign appends its own libresign-ca-id:... entry to the OU field, and the combined value overruns the 64-character ASN.1 limit for organizationalUnitName, failing with string too long.
• A disabled app has no occ commands. If occ list | grep libresign returns nothing, the app is disabled, not missing — occ app:list will still show it under Disabled:. This is what a Nextcloud major upgrade does to an app it thinks is incompatible.
• PHP_MEMORY_LIMIT must be raised above the 512M image default; signing fails opaquely mid-operation otherwise.
• LC_ALL / LANG must be set or the JVM comes up as ANSI_X3.4-1968 and LibreSign warns that accented characters in signer names will be mangled (LibreSign issue #4872).