A spurious invocation carries no action for a reader, so mailing it just
trains them to skip past the subject line -- which defeats the point of the
alert. Send mail only when the unit actually reports failure; spurious
triggers still leave their journald record, so they stay greppable and can
still feed a Graylog rule.
Verified both paths: a spurious trigger leaves /var/log/msmtp.log untouched
and logs alert_mail=skipped reason=spurious, while a genuinely failed unit
still composes mail with the FAILED subject.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.
The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.
The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
esp32-stm32-vcu's scripts/release.sh uses jq to rewrite the protocol
manifest before publishing it to S3. The image did not have it, so the
release aborted at:
./scripts/release.sh: line 61: jq: command not found
The failure mode is nastier than a red job. jq is only reached *after*
the firmware .bin and version.json have already been uploaded, so
clients were served the new build while the git tag, the Gitea release
and the protocol manifest were never written — the repo still showed
the previous release as latest while production served a newer one.
Only jq was missing; aws and sha256sum are already present (verified in
the rebuilt image: jq-1.7, aws-ok, sha256sum-ok).
This surfaced when the skudak firmware jobs were moved into this image
(they previously ran on the default gitea-ci image, which has jq).
The import-only Snipcart key renders empty on normal deploys (import
endpoint 503s); it was injected via EXTRA_VARS just for the historical
refund re-import and is now blanked again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both are pulled on demand by pio run/buildfs; without them baked in, every
ephemeral job container re-downloads them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New opt-in job image localhost/gitea-ci-platformio:7.0.1 (pattern matches the
ESP-IDF image): PlatformIO core 6.1.19 with the espressif32@7.0.1 platform,
xtensa toolchain and Arduino framework pre-baked via a seed project so
ephemeral job containers download nothing. First consumer is
Skudak/esp32-web-interface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the ASCII-safe address transliteration fix (go-fulfillr#24) to prod
after dev verification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the ASCII-safe address transliteration fix (go-fulfillr#24) to the
staging back-office. Prod image tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns build to the prod back-office so the Coupons
admin round-trips the global-sale banner fields on prod. Applied via
`make deploy TAGS=fulfillr`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns bump (coupon banner_label/banner_text) to
the staging back-office so the Coupons admin round-trips the global-sale fields.
Applied via `make deploy TAGS=fulfillr-dev`. Prod tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.0.0 dropped pgvecto.rs support, which crash-looped immich-server
("No vector extension found") and left photos.debyl.io on a white page.
Swap db_image to Immich's bundled postgres image (VectorChord 0.4.3 +
pgvecto.rs 0.2.0) so v3 reindexes the existing embeddings into
VectorChord in place; bump server/ml from v2.7.5 to v3.0.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stripe relink endpoints (stripe-candidates / link-stripe) + ch_ refund support
now live on prod back-office.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.
- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
the alpine image), interval 30s / timeout 5s / retries 3 /
start_period 30s (covers the ~14s HMAC retry backoff), failure_action
restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
build carrying the /health/startup probe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The esp32-stm32-vcu firmware build generates common-yaml headers with
`python3 generate.py`, which needs pyyaml + jinja2. The runner's base Python is
PEP 668 externally-managed (pip install fails) and the IDF venv isn't on PATH in
the docker-exec step shell, so install both as distro packages. Lets firmware
jobs run a plain `python3 generate.py` with no pip and no IDF sourcing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds email_sent/email_failed timeline events for ticket label emails.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ticket refund + replacement-shipment endpoints and guarded transitions.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- remove snipcart_api_key from dev/production config (Snipcart decommissioned
post-migration)
- add review-outreach and cart-recovery schedule_name/schedule_group blocks
(dev + prod) for the EventBridge-driven outreach and cart-recovery jobs
- bump fulfillr image 20260607.0217 -> 20260613.0117
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Containerfile.ci: add python3-yaml + python3-jinja2 and the
gcc-arm-none-eabi / binutils / libnewlib toolchain for embedded builds
- bind-mount the runner's SSH key + known_hosts read-only into each job
container at /root/.ssh so submodule clones over
ssh://git@git.skudak.com:2222 succeed; staged as a dedicated
container_file_t-labelled ci-ssh copy (tasks/user.yml) and allowlisted
via valid_volumes (config.yaml.j2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ollama role and SearXNG container backed FISTO AI responses in the
greg-time Discord bot. greg-time 3.9.6 drops both (plus the Gemini path)
in favor of a single xAI Grok backend, so:
- remove the ollama role and its wiring in deploy_home.yml
- remove the searxng container task, template, and searxng_path default
- gregtime: swap OLLAMA_*/SEARXNG_URL/GEMINI_API_KEY env for XAI_API_KEY,
bump image 3.6.5 -> 3.9.6
- vault: add xai_api_key, drop gemini_api_key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The EasyPost tracker webhook moved to debyltech-api (publicly reachable Lambda);
the fulfillr host is LAN-restricted and no longer hosts it, so the carve-out is
no longer needed. Removes the handle blocks for prod and dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Fulfillr host is IP-restricted, so EasyPost's servers can't reach it. Add a
narrow `handle /webhooks/easypost` before the IP restriction (handle blocks are
mutually exclusive, first match wins) for prod (:9054) and dev (:9055) so the
HMAC-verified tracker webhook is reachable while the rest of the host stays locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lambda packaging steps in some workflows shell out to `zip`; the image
only had `unzip`. Add `zip` alongside it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a second go-fulfillr container (fulfillr-dev) wired to the staging
Turso store + EasyPost/Stripe test keys via dev.json, served at
fulfillr-dev.debyltech.com (Caddy -> :9055), LAN-restricted like prod.
- fulfillr-dev.yml + dev.json.j2: the staging container, volumes, config
- defaults: fulfillr_dev_* vars; prod store URL stubbed off until cutover
- Caddyfile + caddy.yml: fulfillr-dev site block and static mount
- awsddns.yml: Route53 DDNS for the fulfillr-dev hostname
- production.json.j2: add store_database_url/store_auth, rename stripe key
var to fulfillr_stripe_api_key
- vault.yml: dev + store/stripe secrets
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch the act_runners from :host execution to docker:// images backed by
a rootless podman socket under the gitea-runner user, so each job runs in
its own ephemeral container with per-job Go caches. This eliminates the
cross-repo GOMODCACHE/go-build poisoning that forced the debyl runner to
capacity:1.
- deps.yml: enable the rootless --user podman.socket, ensure subuid/subgid,
register gitea_runner_uid; drop the rootful system socket override,
podman-docker and host golang
- images.yml + Containerfile.ci/.espidf: build localhost/gitea-ci and
localhost/gitea-ci-espidf into the runner's rootless image store
- config.yaml.j2: docker:// labels (per-runner overridable), docker_host
-> rootless socket, force_pull false
- act_runner.service.j2: XDG_RUNTIME_DIR + DOCKER_HOST -> user socket
- defaults: uniform capacity:4 (drop the debyl capacity:1 workaround);
esp_idf_version now tags the espressif/idf-based image
- main.yml: import images.yml, drop the host esp-idf install (firmware jobs
use the espressif/idf job container instead)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump fulfillr image to the build with the tickets feature, and add the
tickets_table to the fulfillr production.json config (new debyltech-tickets-prod
DynamoDB table) so the /api/v1/tickets routes register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- add stripe_api_key to fulfillr production.json template
- add restricted Stripe key to ansible vault (encrypted)
- bump fulfillr image to the CI build containing the Stripe endpoints
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bastian de Byl