TrueNAS was power-cycled, the CIFS mounts failed, and systemd never retried
-- mount units are not restarted on failure. SMB came back, nothing
remounted, and immich-server served an empty library for days while its
database still listed 14,181 assets pointing at /mnt/media/originals.
fstab carried no _netdev, no nofail and no automount, so there was no path
back without a human. Now:
x-systemd.automount any access re-attempts the mount; failure stops
being terminal
_netdev / nofail ordered after network-online, dead NAS cannot block boot
soft I/O errors instead of blocking forever, so the
container can be restarted rather than wedging in
uninterruptible sleep
idle-timeout unmount when unused, clearing stale handles
resilienthandles SMB3 rides out brief blips
ansible.posix.mount mounts directly and never starts the generated
.automount unit, leaving the on-access trigger inactive -- enable it
explicitly, or the headline fix silently does nothing.
The containers are systemd USER units while the mounts are SYSTEM units, so
RequiresMountsFor= is unavailable. cifs-watchdog bridges the scopes: checks
health, recovers, and restarts ONLY immich-server (the sole consumer of both
paths; postgres/redis/ML use local volumes).
Two bugs the umount test caught, both worth knowing:
- `ls` cannot test mountedness. An unmounted mount point is an ordinary
empty directory, so ls succeeds and recovery was skipped entirely.
- A drop repaired within a single run leaves prev=healthy, so keying the
restart solely on the stored state skipped it while the container still
held its stale view.
Also moves the SMB password out of /etc/fstab, which is 0644 and was
readable by every local user, into a 0600 credentials file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LibreSign had been silently broken since it was first deployed in
January. Every step of the old before-starting hook ended in `|| echo`,
so six months of failures logged nothing.
LibreSign repair
- Root cause was a stale config_path: a valid OpenSSL root CA existed at
generation 1, a failed CFSSL attempt left an empty generation 2, and
config_path was left pointing at the empty one. Regenerated as
"Skudak LLP" (was the pre-rename "Skudak Rennsport LLP").
- Deleted the hook. Java/PDFtk/jSignPdf live under data/appdata_*, a
persisted volume, so they only ever needed installing once. Install and
verification are now explicit tasks that actually fail.
- PHP_MEMORY_LIMIT 1024M -- the 512M image default fails opaquely
mid-signature. LC_ALL/LANG so the JVM is not ANSI_X3.4-1968.
- signature_render_mode=GRAPHIC_ONLY. Any other mode halves the stamp
width and overlays a name/date block that collides with the drawn mark
and duplicates what our documents already typeset. The value must be
exactly GRAPHIC_ONLY; a bare "GRAPHIC" is accepted by occ, matches no
radio in the UI, and silently reverts to default.
- write_qrcode_on_footer=false, written with --type=boolean because
FooterHandler reads it via getValueBool and the typed appconfig API
does not coerce a string "0". The validation URL text is kept.
- identification_documents=0 -- the default gates signing behind an ID
upload plus admin approval, so signers saw no way to sign.
- shareapi_restrict_user_enumeration_full_match=no, so an email owned by
an existing account can be added as a signer. Root cause is in core
(MailPlugin.php:163), not LibreSign. Do NOT set full_match_email=no --
that disables email signer search entirely.
Mail branding (skudakmail app)
- Two supported extension points, no core patch and no LibreSign fork:
mail_template_class for layout, subjects, button labels and the footer
LibreSign never adds; and a BeforeMessageSent listener to embed the
wordmark as a cid: part so it survives remote-image blocking.
- A third listener adds scoped CSS fixing the signing page being clipped
on iOS Safari (100vh -> 100dvh). Patched upstream too.
- skudakmail-verify.php.j2 asserts all of the above through the real
useTemplate() path and fails the play on drift. Every assertion was
proven to fail when deliberately regressed.
Redis
- memcache.locking was unset, so Nextcloud used DBLockingProvider and
every file lock became a MariaDB write -- the contention behind the
intermittent multi-second stalls. Verified after: db locks static,
redis keys growing.
- requirepass lives in a mounted 0640 conf, not --requirepass, which
would leak it into podman inspect, the systemd unit and ps. The file is
chowned to uid 999 because redis-server does not run as root and the
:ro mount stops the image fixing it itself.
- No maxmemory: cache is evictable, locks are NOT, and evicting a held
lock permits concurrent writers to one file. No persistence either --
a restored RDB could reinstate locks whose owner is long dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not
holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across
16 log streams. Made explicit at 10MiB/keep 3/7d.
Note roll_size et al are subdirectives of `output file`, NOT of `log`.
Getting that wrong does not degrade gracefully: Caddy refuses to start on a
bad config, so every site went down until it was corrected. Worth a
`caddy validate` gate before reload.
journald had no SystemMaxUse and had reached 4 GB, drifting toward its
10%-of-filesystem default (~190 GB on this root). Capped at 500M.
Both are safe to keep short because fluent-bit ships the journal and every
Caddy access log into Graylog -- though note its GELF output has been
erroring for days, which weakens that premise and wants investigating.
The larger find was unrelated to logs: 896 images totalling 59.6 GB with
75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and
5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees
orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added
a weekly timer keeping 30 days so a rollback still needs no rebuild.
Also dropped the decommissioned 6379/tcp redis rule (nothing listening;
Immich's redis is on the shared podman network) and the orphaned nosql, s3
and searxng volume dirs.
Backup log exclusions turned out to be unnecessary: Gitea logs to console
so its log dirs are empty, Nextcloud already excludes its own, BookStack
mounts only uploads, and Caddy is not backed up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Ollama role and SearXNG container backed FISTO AI responses in the
greg-time Discord bot. greg-time 3.9.6 drops both (plus the Gemini path)
in favor of a single xAI Grok backend, so:
- remove the ollama role and its wiring in deploy_home.yml
- remove the searxng container task, template, and searxng_path default
- gregtime: swap OLLAMA_*/SEARXNG_URL/GEMINI_API_KEY env for XAI_API_KEY,
bump image 3.6.5 -> 3.9.6
- vault: add xai_api_key, drop gemini_api_key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a second go-fulfillr container (fulfillr-dev) wired to the staging
Turso store + EasyPost/Stripe test keys via dev.json, served at
fulfillr-dev.debyltech.com (Caddy -> :9055), LAN-restricted like prod.
- fulfillr-dev.yml + dev.json.j2: the staging container, volumes, config
- defaults: fulfillr_dev_* vars; prod store URL stubbed off until cutover
- Caddyfile + caddy.yml: fulfillr-dev site block and static mount
- awsddns.yml: Route53 DDNS for the fulfillr-dev hostname
- production.json.j2: add store_database_url/store_auth, rename stripe key
var to fulfillr_stripe_api_key
- vault.yml: dev + store/stripe secrets
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump fulfillr image to the build with the tickets feature, and add the
tickets_table to the fulfillr production.json config (new debyltech-tickets-prod
DynamoDB table) so the /api/v1/tickets routes register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Bump fulfillr container image from 20260124.0411 to 20260509.1940
(built from go-fulfillr commit 48b9f60 which adds /api/v1/cases
endpoints for the contact-form CRM dashboard).
- Add fulfillr_cases_table default ("debyltech-cases-prod") so the
HasCasesConfig() guard flips on at startup and the cases routes
register.
- Add cases_table to production.json.j2 so it lands in /config inside
the container.
Verified after deploy: GET /api/v1/cases returns the existing test
cases, PATCH succeeds, GSI1PK rewrite works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add n8n container (n8nio/n8n:2.11.3) with Caddy reverse proxy at n8n.debyl.io
- Add --exclude .ssh to cloud backup rsync to prevent overwriting
authorized_keys on TrueNAS backup targets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces old 168-mod collection (3636931465) with new 385-mod collection.
Cleaned BBCode artifacts from mod IDs, updated map folders for 32 maps.
LogCabin retained for player connect/disconnect logging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ollama role for local LLM inference (install, service, models)
- Add searxng container for private search
- Migrate hostname from home.bdebyl.net to home.debyl.io
(inventory, awsddns, zomboid entrypoint, home_server_name)
- Update vault with new secrets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add uptime-kuma-personal container on port 3002
- Add Caddy config for uptime.debyl.io with IP restriction
- Update both uptime-kuma instances to 2.0.2
- Rename debyltech tag from uptime-kuma to uptime-debyltech
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename b42revamp server from "zomboidb42revamp" to "gregboid"
- Remove mod 3238830225 from workshop items
- Replace Real Firearms with B42RainsFirearmsAndGunPartsExpanded4213
- Remove 2788256295/ammomaker mod
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gitea Skudak (git.skudak.com):
- New Gitea instance with PostgreSQL in podman pod under git user
- SSH access via Gitea's built-in SSH server on port 2222
- Registration restricted to @skudak.com emails with email confirmation
- SMTP configured for email delivery
Domain migrations:
- wiki.skudakrennsport.com → wiki.skudak.com (302 redirect)
- cloud.skudakrennsport.com + cloud.skudak.com (dual-domain serving)
- BookStack APP_URL updated to wiki.skudak.com
- Nextcloud trusted_domains updated for cloud.skudak.com
Infrastructure:
- SELinux context for git user container storage (container_file_t)
- Firewall rule for port 2222/tcp (Gitea Skudak SSH)
- Caddy reverse proxy for git.skudak.com
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Gitea container registry login task
- Add graylog.yml with full stack (MongoDB, OpenSearch, Graylog, gelf-proxy)
- Use container image instead of binary for gelf-proxy
- Image tagged from git.debyl.io/debyltech/gelf-proxy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit finalizes the comprehensive migration from nginx + ModSecurity + manual LetsEncrypt
to Caddy v2 with automatic HTTPS. The migration eliminates over 2000 lines of complex
configuration in favor of a single, simplified Caddyfile.
## Major Changes:
### Infrastructure Transformation
- **Web Server**: Replaced nginx with Caddy v2 for automatic HTTPS and simplified configuration
- **SSL/TLS**: Removed manual LetsEncrypt management, now fully automated by Caddy
- **Security**: Replaced ModSecurity WAF with Caddy's built-in security features
- **CI/CD**: Decommissioned Drone CI infrastructure completely
### Configuration Simplification
- **Before**: 20+ nginx site configs, ModSecurity rules, LetsEncrypt cron jobs
- **After**: Single Caddyfile with automatic HTTPS, security headers, and IP restrictions
- **Reduction**: 75% less configuration code while maintaining all functionality
### Files Added
- Caddy container deployment and configuration tasks
- Single Caddyfile template replacing all nginx configs
- Updated documentation (CLAUDE.md, TODO.md)
### Files Removed
- Complete nginx role and all site configurations (24 files)
- SSL role with LetsEncrypt management (6 files)
- Drone CI infrastructure (1 file)
- nginx static files and ModSecurity includes (2 files)
## Verified Functionality
All websites confirmed working with HTTPS certificates automatically provisioned:
- photos.bdebyl.net, parts.bdebyl.net, cloud.bdebyl.net
- wiki.skudakrennsport.com, cloud.skudakrennsport.com
- fulfillr.debyltech.com (with IP restrictions)
- Proper security headers and WebSocket support
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Bastian de Byl