Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs
both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min,
host at 10% charge) and best-effort IPMI power-on when mains returns. The 10%
threshold leans on ignorelb + override.battery.charge.low rather than a custom
poller, because CyberPower asserts its own low-battery flag far too early.
Credentials come from vault vars; nothing sensitive is templated in the clear.
Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which
expects an external caller every ~5 minutes, and nothing was calling. The
personal instance had not run a background job since 2026-05-14 and skudak since
2024-11-20. Consequently trash and file versions never expired, stale chunked
uploads accumulated, calendar reminders never fired, and nextcloud.log was never
rotated -- which quietly made the existing log_rotate_size cap inert. Added a
systemd timer per instance, skipping cleanly when the container is down or in
maintenance so deploy windows don't show up as failed units.
Trash retention on the personal instance: the default "auto" only expires when
disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB
free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional
while still purging early under pressure.
Image bumps:
nextcloud 33.0.0 -> 34.0.2 (both cloud and skudak-cloud)
greg-time-bot 3.9.25 -> 3.10.0
fulfillr 20260723.2044 -> 20260728.2155 (prod and dev)
The fulfillr bump records what is already deployed: both containers were rolled
to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer
update campaign). Committing it keeps the repo from claiming an older tag than
the host is actually running, which would otherwise roll fulfillr backwards on
the next clean-checkout deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A spurious invocation carries no action for a reader, so mailing it just
trains them to skip past the subject line -- which defeats the point of the
alert. Send mail only when the unit actually reports failure; spurious
triggers still leave their journald record, so they stay greppable and can
still feed a Graylog rule.
Verified both paths: a spurious trigger leaves /var/log/msmtp.log untouched
and logs alert_mail=skipped reason=spurious, while a genuinely failed unit
still composes mail with the FAILED subject.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.
The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.
The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The import-only Snipcart key renders empty on normal deploys (import
endpoint 503s); it was injected via EXTRA_VARS just for the historical
refund re-import and is now blanked again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- remove snipcart_api_key from dev/production config (Snipcart decommissioned
post-migration)
- add review-outreach and cart-recovery schedule_name/schedule_group blocks
(dev + prod) for the EventBridge-driven outreach and cart-recovery jobs
- bump fulfillr image 20260607.0217 -> 20260613.0117
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ollama role and SearXNG container backed FISTO AI responses in the
greg-time Discord bot. greg-time 3.9.6 drops both (plus the Gemini path)
in favor of a single xAI Grok backend, so:
- remove the ollama role and its wiring in deploy_home.yml
- remove the searxng container task, template, and searxng_path default
- gregtime: swap OLLAMA_*/SEARXNG_URL/GEMINI_API_KEY env for XAI_API_KEY,
bump image 3.6.5 -> 3.9.6
- vault: add xai_api_key, drop gemini_api_key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The EasyPost tracker webhook moved to debyltech-api (publicly reachable Lambda);
the fulfillr host is LAN-restricted and no longer hosts it, so the carve-out is
no longer needed. Removes the handle blocks for prod and dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Fulfillr host is IP-restricted, so EasyPost's servers can't reach it. Add a
narrow `handle /webhooks/easypost` before the IP restriction (handle blocks are
mutually exclusive, first match wins) for prod (:9054) and dev (:9055) so the
HMAC-verified tracker webhook is reachable while the rest of the host stays locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a second go-fulfillr container (fulfillr-dev) wired to the staging
Turso store + EasyPost/Stripe test keys via dev.json, served at
fulfillr-dev.debyltech.com (Caddy -> :9055), LAN-restricted like prod.
- fulfillr-dev.yml + dev.json.j2: the staging container, volumes, config
- defaults: fulfillr_dev_* vars; prod store URL stubbed off until cutover
- Caddyfile + caddy.yml: fulfillr-dev site block and static mount
- awsddns.yml: Route53 DDNS for the fulfillr-dev hostname
- production.json.j2: add store_database_url/store_auth, rename stripe key
var to fulfillr_stripe_api_key
- vault.yml: dev + store/stripe secrets
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump fulfillr image to the build with the tickets feature, and add the
tickets_table to the fulfillr production.json config (new debyltech-tickets-prod
DynamoDB table) so the /api/v1/tickets routes register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- add stripe_api_key to fulfillr production.json template
- add restricted Stripe key to ansible vault (encrypted)
- bump fulfillr image to the CI build containing the Stripe endpoints
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Bump fulfillr container image from 20260124.0411 to 20260509.1940
(built from go-fulfillr commit 48b9f60 which adds /api/v1/cases
endpoints for the contact-form CRM dashboard).
- Add fulfillr_cases_table default ("debyltech-cases-prod") so the
HasCasesConfig() guard flips on at startup and the cases routes
register.
- Add cases_table to production.json.j2 so it lands in /config inside
the container.
Verified after deploy: GET /api/v1/cases returns the existing test
cases, PATCH succeeds, GSI1PK rewrite works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add n8n container (n8nio/n8n:2.11.3) with Caddy reverse proxy at n8n.debyl.io
- Add --exclude .ssh to cloud backup rsync to prevent overwriting
authorized_keys on TrueNAS backup targets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces old 168-mod collection (3636931465) with new 385-mod collection.
Cleaned BBCode artifacts from mod IDs, updated map folders for 32 maps.
LogCabin retained for player connect/disconnect logging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ollama role for local LLM inference (install, service, models)
- Add searxng container for private search
- Migrate hostname from home.bdebyl.net to home.debyl.io
(inventory, awsddns, zomboid entrypoint, home_server_name)
- Update vault with new secrets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move fluent-bit to common role (systemd service, not a container)
- Move geoip to podman/tasks/data/ (data prep, not a container)
- Remove debyltech tag from geoip (not a debyltech service)
- Fix check_mode for fetch subuid task to enable dry-run mode
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add uptime-kuma-personal container on port 3002
- Add Caddy config for uptime.debyl.io with IP restriction
- Update both uptime-kuma instances to 2.0.2
- Rename debyltech tag from uptime-kuma to uptime-debyltech
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update street2 address to Unit 95
- Add outreach config with DynamoDB tables and SES settings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Deploy systemd path unit that watches for trigger file from Discord
bot and executes world reset script to delete saves and restart server.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Deploy systemd timer that writes zomboid container stats to
zomboid-stats.json every 30 seconds for gregtime to read.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gitea Skudak (git.skudak.com):
- New Gitea instance with PostgreSQL in podman pod under git user
- SSH access via Gitea's built-in SSH server on port 2222
- Registration restricted to @skudak.com emails with email confirmation
- SMTP configured for email delivery
Domain migrations:
- wiki.skudakrennsport.com → wiki.skudak.com (302 redirect)
- cloud.skudakrennsport.com + cloud.skudak.com (dual-domain serving)
- BookStack APP_URL updated to wiki.skudak.com
- Nextcloud trusted_domains updated for cloud.skudak.com
Infrastructure:
- SELinux context for git user container storage (container_file_t)
- Firewall rule for port 2222/tcp (Gitea Skudak SSH)
- Caddy reverse proxy for git.skudak.com
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Configure Caddy to send CORS headers for /api/* endpoints
- Allow all origins (*) since API is already IP-restricted to local network
- Handle preflight OPTIONS requests properly
- Update container versions (immich, fulfillr)
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit finalizes the comprehensive migration from nginx + ModSecurity + manual LetsEncrypt
to Caddy v2 with automatic HTTPS. The migration eliminates over 2000 lines of complex
configuration in favor of a single, simplified Caddyfile.
## Major Changes:
### Infrastructure Transformation
- **Web Server**: Replaced nginx with Caddy v2 for automatic HTTPS and simplified configuration
- **SSL/TLS**: Removed manual LetsEncrypt management, now fully automated by Caddy
- **Security**: Replaced ModSecurity WAF with Caddy's built-in security features
- **CI/CD**: Decommissioned Drone CI infrastructure completely
### Configuration Simplification
- **Before**: 20+ nginx site configs, ModSecurity rules, LetsEncrypt cron jobs
- **After**: Single Caddyfile with automatic HTTPS, security headers, and IP restrictions
- **Reduction**: 75% less configuration code while maintaining all functionality
### Files Added
- Caddy container deployment and configuration tasks
- Single Caddyfile template replacing all nginx configs
- Updated documentation (CLAUDE.md, TODO.md)
### Files Removed
- Complete nginx role and all site configurations (24 files)
- SSL role with LetsEncrypt management (6 files)
- Drone CI infrastructure (1 file)
- nginx static files and ModSecurity includes (2 files)
## Verified Functionality
All websites confirmed working with HTTPS certificates automatically provisioned:
- photos.bdebyl.net, parts.bdebyl.net, cloud.bdebyl.net
- wiki.skudakrennsport.com, cloud.skudakrennsport.com
- fulfillr.debyltech.com (with IP restrictions)
- Proper security headers and WebSocket support
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Bastian de Byl