A host-to-iDrive S3 stage was added here and is now removed. It would
have written the same data into the same `backup-all` bucket that the
TrueNAS cloud-sync task already fills -- duplicate storage, two writers
to one prefix, for no additional coverage.
Offsite to business-owned storage was already solved: the rsync feeds
/mnt/glacier/skudakcloud and TrueNAS cloud-syncs that to Skudak's own
iDrive e2 account. The earlier note in skudak/cloud.yml proposed adding
S3 *and then dropping the rsync* -- replacement, not addition -- and
building both was a misreading of it.
If offsite is ever moved onto this host it must REPLACE the rsync, not
run beside it. Settle first whether the TrueNAS -> iDrive leg is
independently verifiable; keeping this chain means trusting it.
Also removes the now-orphaned /etc/backup_s3 credential file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not
holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across
16 log streams. Made explicit at 10MiB/keep 3/7d.
Note roll_size et al are subdirectives of `output file`, NOT of `log`.
Getting that wrong does not degrade gracefully: Caddy refuses to start on a
bad config, so every site went down until it was corrected. Worth a
`caddy validate` gate before reload.
journald had no SystemMaxUse and had reached 4 GB, drifting toward its
10%-of-filesystem default (~190 GB on this root). Capped at 500M.
Both are safe to keep short because fluent-bit ships the journal and every
Caddy access log into Graylog -- though note its GELF output has been
erroring for days, which weakens that premise and wants investigating.
The larger find was unrelated to logs: 896 images totalling 59.6 GB with
75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and
5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees
orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added
a weekly timer keeping 30 days so a rollback still needs no rebuild.
Also dropped the decommissioned 6379/tcp redis rule (nothing listening;
Immich's redis is on the shared podman network) and the orphaned nosql, s3
and searxng volume dirs.
Backup log exclusions turned out to be unnecessary: Gitea logs to console
so its log dirs are empty, Nextcloud already excludes its own, BookStack
mounts only uploads, and Caddy is not backed up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs
both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min,
host at 10% charge) and best-effort IPMI power-on when mains returns. The 10%
threshold leans on ignorelb + override.battery.charge.low rather than a custom
poller, because CyberPower asserts its own low-battery flag far too early.
Credentials come from vault vars; nothing sensitive is templated in the clear.
Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which
expects an external caller every ~5 minutes, and nothing was calling. The
personal instance had not run a background job since 2026-05-14 and skudak since
2024-11-20. Consequently trash and file versions never expired, stale chunked
uploads accumulated, calendar reminders never fired, and nextcloud.log was never
rotated -- which quietly made the existing log_rotate_size cap inert. Added a
systemd timer per instance, skipping cleanly when the container is down or in
maintenance so deploy windows don't show up as failed units.
Trash retention on the personal instance: the default "auto" only expires when
disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB
free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional
while still purging early under pressure.
Image bumps:
nextcloud 33.0.0 -> 34.0.2 (both cloud and skudak-cloud)
greg-time-bot 3.9.25 -> 3.10.0
fulfillr 20260723.2044 -> 20260728.2155 (prod and dev)
The fulfillr bump records what is already deployed: both containers were rolled
to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer
update campaign). Committing it keeps the repo from claiming an older tag than
the host is actually running, which would otherwise roll fulfillr backwards on
the next clean-checkout deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A spurious invocation carries no action for a reader, so mailing it just
trains them to skip past the subject line -- which defeats the point of the
alert. Send mail only when the unit actually reports failure; spurious
triggers still leave their journald record, so they stay greppable and can
still feed a Graylog rule.
Verified both paths: a spurious trigger leaves /var/log/msmtp.log untouched
and logs alert_mail=skipped reason=spurious, while a genuinely failed unit
still composes mail with the FAILED subject.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.
The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.
The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Referenced by roles/common/templates/msmtp/msmtprc.j2 to authenticate
outbound alert mail against the OpenSRS relay. Without this committed a
fresh checkout cannot render /etc/msmtprc.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
esp32-stm32-vcu's scripts/release.sh uses jq to rewrite the protocol
manifest before publishing it to S3. The image did not have it, so the
release aborted at:
./scripts/release.sh: line 61: jq: command not found
The failure mode is nastier than a red job. jq is only reached *after*
the firmware .bin and version.json have already been uploaded, so
clients were served the new build while the git tag, the Gitea release
and the protocol manifest were never written — the repo still showed
the previous release as latest while production served a newer one.
Only jq was missing; aws and sha256sum are already present (verified in
the rebuilt image: jq-1.7, aws-ok, sha256sum-ok).
This surfaced when the skudak firmware jobs were moved into this image
(they previously ran on the default gitea-ci image, which has jq).
The import-only Snipcart key renders empty on normal deploys (import
endpoint 503s); it was injected via EXTRA_VARS just for the historical
refund re-import and is now blanked again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both are pulled on demand by pio run/buildfs; without them baked in, every
ephemeral job container re-downloads them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New opt-in job image localhost/gitea-ci-platformio:7.0.1 (pattern matches the
ESP-IDF image): PlatformIO core 6.1.19 with the espressif32@7.0.1 platform,
xtensa toolchain and Arduino framework pre-baked via a seed project so
ephemeral job containers download nothing. First consumer is
Skudak/esp32-web-interface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the ASCII-safe address transliteration fix (go-fulfillr#24) to prod
after dev verification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the ASCII-safe address transliteration fix (go-fulfillr#24) to the
staging back-office. Prod image tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns build to the prod back-office so the Coupons
admin round-trips the global-sale banner fields on prod. Applied via
`make deploy TAGS=fulfillr`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns bump (coupon banner_label/banner_text) to
the staging back-office so the Coupons admin round-trips the global-sale fields.
Applied via `make deploy TAGS=fulfillr-dev`. Prod tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.0.0 dropped pgvecto.rs support, which crash-looped immich-server
("No vector extension found") and left photos.debyl.io on a white page.
Swap db_image to Immich's bundled postgres image (VectorChord 0.4.3 +
pgvecto.rs 0.2.0) so v3 reindexes the existing embeddings into
VectorChord in place; bump server/ml from v2.7.5 to v3.0.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stripe relink endpoints (stripe-candidates / link-stripe) + ch_ refund support
now live on prod back-office.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.
- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
the alpine image), interval 30s / timeout 5s / retries 3 /
start_period 30s (covers the ~14s HMAC retry backoff), failure_action
restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
build carrying the /health/startup probe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The esp32-stm32-vcu firmware build generates common-yaml headers with
`python3 generate.py`, which needs pyyaml + jinja2. The runner's base Python is
PEP 668 externally-managed (pip install fails) and the IDF venv isn't on PATH in
the docker-exec step shell, so install both as distro packages. Lets firmware
jobs run a plain `python3 generate.py` with no pip and no IDF sourcing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds email_sent/email_failed timeline events for ticket label emails.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ticket refund + replacement-shipment endpoints and guarded transitions.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- remove snipcart_api_key from dev/production config (Snipcart decommissioned
post-migration)
- add review-outreach and cart-recovery schedule_name/schedule_group blocks
(dev + prod) for the EventBridge-driven outreach and cart-recovery jobs
- bump fulfillr image 20260607.0217 -> 20260613.0117
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Containerfile.ci: add python3-yaml + python3-jinja2 and the
gcc-arm-none-eabi / binutils / libnewlib toolchain for embedded builds
- bind-mount the runner's SSH key + known_hosts read-only into each job
container at /root/.ssh so submodule clones over
ssh://git@git.skudak.com:2222 succeed; staged as a dedicated
container_file_t-labelled ci-ssh copy (tasks/user.yml) and allowlisted
via valid_volumes (config.yaml.j2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ollama role and SearXNG container backed FISTO AI responses in the
greg-time Discord bot. greg-time 3.9.6 drops both (plus the Gemini path)
in favor of a single xAI Grok backend, so:
- remove the ollama role and its wiring in deploy_home.yml
- remove the searxng container task, template, and searxng_path default
- gregtime: swap OLLAMA_*/SEARXNG_URL/GEMINI_API_KEY env for XAI_API_KEY,
bump image 3.6.5 -> 3.9.6
- vault: add xai_api_key, drop gemini_api_key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The EasyPost tracker webhook moved to debyltech-api (publicly reachable Lambda);
the fulfillr host is LAN-restricted and no longer hosts it, so the carve-out is
no longer needed. Removes the handle blocks for prod and dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Fulfillr host is IP-restricted, so EasyPost's servers can't reach it. Add a
narrow `handle /webhooks/easypost` before the IP restriction (handle blocks are
mutually exclusive, first match wins) for prod (:9054) and dev (:9055) so the
HMAC-verified tracker webhook is reachable while the rest of the host stays locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bastian de Byl
bastian