deploy_home/ansible/roles/podman/README.md

# podman role

Container orchestration for the home server. Containers are defined under
`tasks/containers/{base,home,skudak,debyltech}/` and wired from
`tasks/main.yml`, which is where every image tag is pinned.

## Backups

Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one
backup engine: `tasks/containers/cloud-backup.yml` plus
`templates/nextcloud/cloud-backup.sh.j2`. Each instance includes it with its own
vars, producing `/usr/local/bin/<name>-backup.sh` and a systemd timer.

Stages, in order — the ordering is deliberate, see the comments in the template:

1. **Database dump** inside the container (mariadb / mysql / postgres branches),
   gzipped to `/var/backups/nextcloud/<name>/db/`. Promoted over yesterday's dump
   only after passing `gzip -t` **and** a completion-trailer grep.
2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used.
3. **rsync** of the data tree, config, and db dumps to TrueNAS.

Failures raise `status=failed` on the `nextcloud-backup` syslog tag, which an
external Graylog rule matches to send mail. Do not rename that tag: it is shared
by every instance including Gitea, and renaming it here silently stops alerting
for all of them.


## Restore

**Untested backups are not a control.** Rehearse this into a scratch location
before you need it, and record the date you last did.

### 1. Database

Dumps live on the host at `/var/backups/nextcloud/<name>/db/<name>-YYYYMMDD.sql.gz`
and on TrueNAS at `<remote_path>/_backup/db/`. TrueNAS in turn cloud-syncs
`/mnt/glacier/skudakcloud` to Skudak's own iDrive e2 bucket, so a third copy
exists there — but restoring from it means going through the TrueNAS console,
not this host.

Verify the dump before trusting it:

```bash
gzip -t <name>-YYYYMMDD.sql.gz
gunzip -c <name>-YYYYMMDD.sql.gz | tail -c 512   # expect the completion trailer
```

Replay into the running database container. MariaDB/MySQL:

```bash
sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
  | podman exec -i <db_container> sh -c \
    "exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"'
```

Postgres dumps are taken with `--clean --if-exists --no-owner`, so they replay
into an existing database:

```bash
sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
  | podman exec -i <db_container> sh -c \
    "exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"'
```

### 2. Data tree

```bash
# from TrueNAS
rsync -az -e "ssh -i /etc/ssh/backup_keys/<name>" \
  <ssh_user>@truenas.localdomain:<remote_path>/ <volumes>/<instance>/data/

```

Then fix ownership — the containers run as uid 33 inside a rootless userns:

```bash
sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 <volumes>/<instance>/data'
```

### 3. Reconcile

```bash
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --on'
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ files:scan --all'
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --off'
```

A DB snapshot slightly **older** than the files degrades to "files the app has
not indexed yet" and is repaired by `files:scan`. A DB snapshot **newer** than
the files references blobs that were never backed up, which surfaces as broken
shares and dead file entries — this is why the dump runs first.

### 4. LibreSign-specific

The signing CA lives in the data tree at
`data/appdata_*/libresign/pki/<instance>_<n>_openssl/`, so a data-tree restore
brings it back with everything else. After restoring, confirm it:

```bash
sudo -H -u podman bash -c 'cd; podman exec -u www-data skudak-cloud php occ libresign:configure:check'
```

Every check must report `success`. If `openssl-configure` reports an error, the
`certificate_engine` / `config_path` app config is pointing somewhere without a
CA — see the guarded generate task in `tasks/containers/skudak/cloud.yml`.
**Do not** simply re-run `libresign:configure:openssl` on a restored instance
without understanding why: it mints a *new* root CA and invalidates the trust
chain on every document already signed under the old one.

## LibreSign

Deployed on `skudak-cloud` only. LibreSign 14.1.0 requires Nextcloud server
`>=34.0.0,<35.0.0`, which the pinned `nextcloud:34.0.2-apache` satisfies. If the
Nextcloud tag is bumped to 35, LibreSign must be held or upgraded in step — the
two instances are pinned independently in `tasks/main.yml`, so `skudak-cloud`
can lag `cloud` if needed.

Dependency split, which drives what survives a container recreate:

| Component | Location | Survives recreate? |
|---|---|---|
| Java (JRE 21), PDFtk, jSignPdf | `data/appdata_*/libresign/` | **Yes** — persisted volume |
| Root CA / PKI | `data/appdata_*/libresign/pki/` | **Yes** — persisted volume |
| poppler-utils, ghostscript | `/usr` in the image | **No** — reinstalled by Ansible each run |

Certificate engine is **OpenSSL**, not CFSSL. CFSSL is the more common source of
LibreSign setup failures and buys nothing at this scale.

### Gotchas

- **Do not pass `--ou`** to `libresign:configure:openssl`. LibreSign appends its
  own `libresign-ca-id:...` entry to the OU field, and the combined value
  overruns the 64-character ASN.1 limit for `organizationalUnitName`, failing
  with `string too long`.
- **A disabled app has no `occ` commands.** If `occ list | grep libresign`
  returns nothing, the app is disabled, not missing — `occ app:list` will still
  show it under `Disabled:`. This is what a Nextcloud major upgrade does to an
  app it thinks is incompatible.
- `PHP_MEMORY_LIMIT` must be raised above the 512M image default; signing fails
  opaquely mid-operation otherwise.
- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and
  LibreSign warns that accented characters in signer names will be mangled
  (LibreSign issue #4872).