Bastian de BylandClaude Opus 5.5
0cc4c1460e
Nothing was group-restricted, so customers saw Dashboard, Photos, Office and the rest, plus Nextcloud's first-run and promo apps. - Disable for everyone: firstrunwizard, recommendations, related_resources, weather_status, survey_client, support, app_api, contactsinteraction, photos. A refused disable now fails the play (occ exits 0 on "can't be disabled"). - Restrict dashboard and office to staff. defaultapp=dashboard,files so staff land on the dashboard and customers fall through to Files. - libresign groups_request_sign pinned to staff and asserted in verify. LibreSign itself is deliberately NOT group-restricted: that also blocks anonymous requests and would break public signing links. - profile.enabled=false; lookup_server="" (lookup_server_connector can't be disabled). - README: what customers can open. Checked as a probe customer: apps=files,activity,libresign,text,viewer, lands in Files, can't request signatures; staff land on Dashboard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
247 lines
11 KiB
Markdown
247 lines
11 KiB
Markdown
# podman role
|
|
|
|
Container orchestration for the home server. Containers are defined under
|
|
`tasks/containers/{base,home,skudak,debyltech}/` and wired from
|
|
`tasks/main.yml`, which is where every image tag is pinned.
|
|
|
|
## Backups
|
|
|
|
Every Nextcloud-style instance (Nextcloud, Gitea, BookStack, partsy) shares one
|
|
backup engine: `tasks/containers/cloud-backup.yml` plus
|
|
`templates/nextcloud/cloud-backup.sh.j2`. Each instance includes it with its own
|
|
vars, producing `/usr/local/bin/<name>-backup.sh` and a systemd timer.
|
|
|
|
Stages, in order — the ordering is deliberate, see the comments in the template:
|
|
|
|
1. **Database dump** inside the container (mariadb / mysql / postgres branches),
|
|
gzipped to `/var/backups/nextcloud/<name>/db/`. Promoted over yesterday's dump
|
|
only after passing `gzip -t` **and** a completion-trailer grep.
|
|
2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used.
|
|
3. **rsync** of the data tree, config, and db dumps to TrueNAS.
|
|
|
|
Failures raise `status=failed` on the `nextcloud-backup` syslog tag. The mail
|
|
itself is sent by the unit's own `OnFailure=` handler
|
|
(`templates/nextcloud/nextcloud-backup-alert.sh.j2`) straight through
|
|
`sendmail`, so alerting does **not** depend on Graylog and is unaffected by
|
|
`graylog_enabled` being off. A Graylog rule matching the same tag is a
|
|
secondary, dashboard-side copy of that signal.
|
|
|
|
Do not rename that tag: it is shared by every instance including Gitea, and
|
|
renaming it here silently stops the Graylog-side alerting for all of them.
|
|
|
|
|
|
## Restore
|
|
|
|
**Untested backups are not a control.** Rehearse this into a scratch location
|
|
before you need it, and record the date you last did.
|
|
|
|
### 1. Database
|
|
|
|
Dumps live on the host at `/var/backups/nextcloud/<name>/db/<name>-YYYYMMDD.sql.gz`
|
|
and on TrueNAS at `<remote_path>/_backup/db/`. TrueNAS in turn cloud-syncs
|
|
each dataset offsite, so a third copy exists there — but restoring from it
|
|
means going through the TrueNAS console, not this host:
|
|
|
|
| Dataset | Offsite |
|
|
|---|---|
|
|
| `skudakcloud`, `skudakapps`, `skudakgit` | Skudak's own iDrive e2 bucket (excluded from the personal task) |
|
|
| `nextcloud`, `gitea`, `debyltechcloud` | Personal iDrive e2 bucket, via the "iDrive E2 Backup" task over `/mnt/glacier` |
|
|
|
|
Verify the dump before trusting it:
|
|
|
|
```bash
|
|
gzip -t <name>-YYYYMMDD.sql.gz
|
|
gunzip -c <name>-YYYYMMDD.sql.gz | tail -c 512 # expect the completion trailer
|
|
```
|
|
|
|
Replay into the running database container. MariaDB/MySQL:
|
|
|
|
```bash
|
|
sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
|
|
| podman exec -i <db_container> sh -c \
|
|
"exec env MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mariadb -u root \$MYSQL_DATABASE"'
|
|
```
|
|
|
|
Postgres dumps are taken with `--clean --if-exists --no-owner`, so they replay
|
|
into an existing database:
|
|
|
|
```bash
|
|
sudo -H -u podman bash -c 'cd; gunzip -c /path/to/dump.sql.gz \
|
|
| podman exec -i <db_container> sh -c \
|
|
"exec env PGPASSWORD=\$POSTGRES_PASSWORD psql -U \$POSTGRES_USER \$POSTGRES_DB"'
|
|
```
|
|
|
|
### 2. Data tree
|
|
|
|
```bash
|
|
# from TrueNAS
|
|
rsync -az -e "ssh -i /etc/ssh/backup_keys/<name>" \
|
|
<ssh_user>@truenas.localdomain:<remote_path>/ <volumes>/<instance>/data/
|
|
|
|
```
|
|
|
|
Then fix ownership — the containers run as uid 33 inside a rootless userns:
|
|
|
|
```bash
|
|
sudo -H -u podman bash -c 'cd; podman unshare chown -R 33:33 <volumes>/<instance>/data'
|
|
```
|
|
|
|
### 3. Reconcile
|
|
|
|
```bash
|
|
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --on'
|
|
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ files:scan --all'
|
|
sudo -H -u podman bash -c 'cd; podman exec -u www-data <container> php occ maintenance:mode --off'
|
|
```
|
|
|
|
A DB snapshot slightly **older** than the files degrades to "files the app has
|
|
not indexed yet" and is repaired by `files:scan`. A DB snapshot **newer** than
|
|
the files references blobs that were never backed up, which surfaces as broken
|
|
shares and dead file entries — this is why the dump runs first.
|
|
|
|
### 4. LibreSign-specific
|
|
|
|
The signing CA lives in the data tree at
|
|
`data/appdata_*/libresign/pki/<instance>_<n>_openssl/`, so a data-tree restore
|
|
brings it back with everything else. After restoring, confirm it:
|
|
|
|
```bash
|
|
sudo -H -u podman bash -c 'cd; podman exec -u www-data <skudak-cloud|debyltech-cloud> php occ libresign:configure:check'
|
|
```
|
|
|
|
Every check must report `success`. If `openssl-configure` reports an error, the
|
|
`certificate_engine` / `config_path` app config is pointing somewhere without a
|
|
CA — see the guarded generate task in `tasks/containers/{skudak,debyltech}/cloud.yml`.
|
|
**Do not** simply re-run `libresign:configure:openssl` on a restored instance
|
|
without understanding why: it mints a *new* root CA and invalidates the trust
|
|
chain on every document already signed under the old one.
|
|
|
|
## LibreSign
|
|
|
|
Deployed on `skudak-cloud` and `debyltech-cloud`. LibreSign 14.1.0 requires
|
|
Nextcloud server `>=34.0.0,<35.0.0`, which the pinned `nextcloud:34.0.3-apache`
|
|
satisfies. If the Nextcloud tag is bumped to 35, LibreSign must be held or
|
|
upgraded in step — each instance is pinned independently in `tasks/main.yml`,
|
|
so the LibreSign instances can lag `cloud` if needed. The branding apps
|
|
(`files/skudakmail`, `files/debyltechmail`) pin `max-version="34"` too and
|
|
must be bumped alongside.
|
|
|
|
Dependency split, which drives what survives a container recreate:
|
|
|
|
| Component | Location | Survives recreate? |
|
|
|---|---|---|
|
|
| Java (JRE 21), PDFtk, jSignPdf | `data/appdata_*/libresign/` | **Yes** — persisted volume |
|
|
| Root CA / PKI | `data/appdata_*/libresign/pki/` | **Yes** — persisted volume |
|
|
| poppler-utils, ghostscript | `/usr` in the image | **No** — reinstalled by Ansible each run |
|
|
|
|
Certificate engine is **OpenSSL**, not CFSSL. CFSSL is the more common source of
|
|
LibreSign setup failures and buys nothing at this scale.
|
|
|
|
### Gotchas
|
|
|
|
- **Do not pass `--ou`** to `libresign:configure:openssl`. LibreSign appends its
|
|
own `libresign-ca-id:...` entry to the OU field, and the combined value
|
|
overruns the 64-character ASN.1 limit for `organizationalUnitName`, failing
|
|
with `string too long`.
|
|
- **A disabled app has no `occ` commands.** If `occ list | grep libresign`
|
|
returns nothing, the app is disabled, not missing — `occ app:list` will still
|
|
show it under `Disabled:`. This is what a Nextcloud major upgrade does to an
|
|
app it thinks is incompatible.
|
|
- `PHP_MEMORY_LIMIT` must be raised above the 512M image default; signing fails
|
|
opaquely mid-operation otherwise.
|
|
- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and
|
|
LibreSign warns that accented characters in signer names will be mangled
|
|
(LibreSign issue #4872).
|
|
|
|
|
|
## cloud.debyltech.com accounts
|
|
|
|
Registration is off, so every account is created by an admin in
|
|
Settings → Accounts. The isolation policy in
|
|
`tasks/containers/debyltech/cloud.yml` is enforced on every deploy and checked
|
|
by the verify script.
|
|
|
|
**Staff**: add to the `admin` group (the only group allowed to share) and to
|
|
`cloud_debyltech_staff_users` in `defaults/main.yml`, so the next deploy
|
|
exempts them from the 0 B default quota. Until then, set the account's quota
|
|
to *Unlimited* by hand.
|
|
|
|
**Customer**:
|
|
1. Create a group per customer, e.g. `Customer - Acme`. Use a consistent
|
|
prefix: autocomplete is off, so you share by typing the **exact** group
|
|
name.
|
|
2. Create the account with only their email filled in and no password, in
|
|
that group. Nextcloud sends a branded welcome email with a set-password
|
|
link.
|
|
3. Share a folder (for example `Customers/Acme`) with the group. It defaults
|
|
to **View only**; choose *Allow editing* only if they should upload.
|
|
|
|
What a customer gets:
|
|
- read-only access to what's shared with them
|
|
- no personal storage (0 B quota)
|
|
- no sharing or public links
|
|
- no view of other accounts or groups: the share search and contacts menu
|
|
return nothing
|
|
|
|
What they can open: Files, Activity, and LibreSign for signing only.
|
|
Dashboard and Office are staff-only. Promotional or directory-style apps
|
|
(first-run wizard, recommendations, weather, Photos, contacts interaction,
|
|
lookup server, ...) are disabled for everyone. Only staff can *request*
|
|
signatures.
|
|
|
|
Signature requests to customers need no account: use LibreSign with their
|
|
email address. LibreSign stays enabled for all accounts on purpose.
|
|
Restricting an app to a group also blocks visitors who aren't logged in,
|
|
which would break the public signing links.
|
|
|
|
## Logging
|
|
|
|
Every container runs with `log_driver=journald`, so container stdout lands in
|
|
the host journal, capped at 500M by `roles/common/templates/journald-size.conf.j2`
|
|
(about 25 days at the current rate). `journalctl CONTAINER_NAME=<name>` is the
|
|
day-to-day way to read it.
|
|
|
|
On top of that sits an optional Graylog stack — `graylog`, `graylog-opensearch`,
|
|
`graylog-mongo`, fed by a host `fluent-bit` service that tails the journal and
|
|
ships GELF to `127.0.0.1:12202`, enriched with the MaxMind GeoIP database, and
|
|
configured over the REST API by the separate `graylog-config` role.
|
|
|
|
**It is off.** The switch is `graylog_enabled` in
|
|
`inventories/home/hosts.yml`, and it gates all five of those pieces at once:
|
|
|
|
| `graylog_enabled` | effect |
|
|
| --- | --- |
|
|
| `true` | stack deployed, fluent-bit shipping, GeoIP downloaded, `graylog-config` runs, `logs.debyl.io` proxies to the UI and `/gelf` |
|
|
| `false` | containers removed and their systemd user units disabled and deleted, fluent-bit stopped and disabled, GeoIP and `graylog-config` skipped, `logs.debyl.io` answers 503 |
|
|
|
|
It was turned off because the cost/benefit is bad on a 4-core box: two JVMs plus
|
|
MongoDB held ~1.6 GB resident and ~3% of the CPU continuously to store roughly
|
|
3k messages a day — about 28 MB of real log data across four live indices — all
|
|
of which journald already keeps for longer.
|
|
|
|
Turning it back on is `graylog_enabled: true` plus `make deploy TAGS=graylog`.
|
|
Nothing is destroyed by the off path: the volumes under `{{ graylog_path }}`
|
|
keep the indices, the Mongo database holding streams/pipelines/dashboards, and
|
|
the node-id file, so the stack comes back with its configuration intact.
|
|
|
|
Two things genuinely stop while it is off, both by design:
|
|
|
|
- **Search and dashboards.** The logs still exist in journald; the query
|
|
interface over them does not.
|
|
- **External GELF ingest.** The AWS Lambda that POSTs to `logs.debyl.io/gelf`
|
|
for the `debyltech-api` stream gets a 503. Those events are dropped, not
|
|
queued — nothing else records them.
|
|
|
|
### Caddy config reloads
|
|
|
|
The `reload caddy` handler deliberately reads `/config/Caddyfile`, not the
|
|
`/etc/caddy/Caddyfile` the container starts from, even though both are the same
|
|
host file. `/etc/caddy/Caddyfile` is a **single-file** bind mount, which podman
|
|
binds by inode, and Ansible's `template` module writes a temp file and renames
|
|
it into place — so every deploy gives the host file a new inode while the
|
|
container keeps seeing the one it was created with. Reloading from that path
|
|
silently re-applied the previous config; changes only landed when something
|
|
recreated the container. `{{ caddy_path }}/config` is also bind-mounted as a
|
|
*directory* at `/config`, and directory mounts resolve names at `open()` time,
|
|
so `/config/Caddyfile` is always the file Ansible just wrote.
|