LibreSign had been silently broken since it was first deployed in
January. Every step of the old before-starting hook ended in `|| echo`,
so six months of failures logged nothing.
LibreSign repair
- Root cause was a stale config_path: a valid OpenSSL root CA existed at
generation 1, a failed CFSSL attempt left an empty generation 2, and
config_path was left pointing at the empty one. Regenerated as
"Skudak LLP" (was the pre-rename "Skudak Rennsport LLP").
- Deleted the hook. Java/PDFtk/jSignPdf live under data/appdata_*, a
persisted volume, so they only ever needed installing once. Install and
verification are now explicit tasks that actually fail.
- PHP_MEMORY_LIMIT 1024M -- the 512M image default fails opaquely
mid-signature. LC_ALL/LANG so the JVM is not ANSI_X3.4-1968.
- signature_render_mode=GRAPHIC_ONLY. Any other mode halves the stamp
width and overlays a name/date block that collides with the drawn mark
and duplicates what our documents already typeset. The value must be
exactly GRAPHIC_ONLY; a bare "GRAPHIC" is accepted by occ, matches no
radio in the UI, and silently reverts to default.
- write_qrcode_on_footer=false, written with --type=boolean because
FooterHandler reads it via getValueBool and the typed appconfig API
does not coerce a string "0". The validation URL text is kept.
- identification_documents=0 -- the default gates signing behind an ID
upload plus admin approval, so signers saw no way to sign.
- shareapi_restrict_user_enumeration_full_match=no, so an email owned by
an existing account can be added as a signer. Root cause is in core
(MailPlugin.php:163), not LibreSign. Do NOT set full_match_email=no --
that disables email signer search entirely.
Mail branding (skudakmail app)
- Two supported extension points, no core patch and no LibreSign fork:
mail_template_class for layout, subjects, button labels and the footer
LibreSign never adds; and a BeforeMessageSent listener to embed the
wordmark as a cid: part so it survives remote-image blocking.
- A third listener adds scoped CSS fixing the signing page being clipped
on iOS Safari (100vh -> 100dvh). Patched upstream too.
- skudakmail-verify.php.j2 asserts all of the above through the real
useTemplate() path and fails the play on drift. Every assertion was
proven to fail when deliberately regressed.
Redis
- memcache.locking was unset, so Nextcloud used DBLockingProvider and
every file lock became a MariaDB write -- the contention behind the
intermittent multi-second stalls. Verified after: db locks static,
redis keys growing.
- requirepass lives in a mounted 0640 conf, not --requirepass, which
would leak it into podman inspect, the systemd unit and ps. The file is
chowned to uid 999 because redis-server does not run as root and the
:ro mount stops the image fixing it itself.
- No maxmemory: cache is evictable, locks are NOT, and evicting a held
lock permits concurrent writers to one file. No persistence either --
a restored RDB could reinstate locks whose owner is long dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs
both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min,
host at 10% charge) and best-effort IPMI power-on when mains returns. The 10%
threshold leans on ignorelb + override.battery.charge.low rather than a custom
poller, because CyberPower asserts its own low-battery flag far too early.
Credentials come from vault vars; nothing sensitive is templated in the clear.
Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which
expects an external caller every ~5 minutes, and nothing was calling. The
personal instance had not run a background job since 2026-05-14 and skudak since
2024-11-20. Consequently trash and file versions never expired, stale chunked
uploads accumulated, calendar reminders never fired, and nextcloud.log was never
rotated -- which quietly made the existing log_rotate_size cap inert. Added a
systemd timer per instance, skipping cleanly when the container is down or in
maintenance so deploy windows don't show up as failed units.
Trash retention on the personal instance: the default "auto" only expires when
disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB
free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional
while still purging early under pressure.
Image bumps:
nextcloud 33.0.0 -> 34.0.2 (both cloud and skudak-cloud)
greg-time-bot 3.9.25 -> 3.10.0
fulfillr 20260723.2044 -> 20260728.2155 (prod and dev)
The fulfillr bump records what is already deployed: both containers were rolled
to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer
update campaign). Committing it keeps the repo from claiming an older tag than
the host is actually running, which would otherwise roll fulfillr backwards on
the next clean-checkout deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSH keys moved to /etc/ssh/backup_keys/ (ssh_home_t) and backup scripts
to /usr/local/bin/ (bin_t) to fix SELinux denials - container_file_t
context blocked rsync from exec'ing ssh. Also fixes skudak key path
mismatch (was truenas_skudak, key deployed as truenas_skudak-cloud).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces old 168-mod collection (3636931465) with new 385-mod collection.
Cleaned BBCode artifacts from mod IDs, updated map folders for 32 maps.
LogCabin retained for player connect/disconnect logging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gitea Skudak (git.skudak.com):
- New Gitea instance with PostgreSQL in podman pod under git user
- SSH access via Gitea's built-in SSH server on port 2222
- Registration restricted to @skudak.com emails with email confirmation
- SMTP configured for email delivery
Domain migrations:
- wiki.skudakrennsport.com → wiki.skudak.com (302 redirect)
- cloud.skudakrennsport.com + cloud.skudak.com (dual-domain serving)
- BookStack APP_URL updated to wiki.skudak.com
- Nextcloud trusted_domains updated for cloud.skudak.com
Infrastructure:
- SELinux context for git user container storage (container_file_t)
- Firewall rule for port 2222/tcp (Gitea Skudak SSH)
- Caddy reverse proxy for git.skudak.com
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Bastian de Byl