mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.
The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.
The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Referenced by roles/common/templates/msmtp/msmtprc.j2 to authenticate
outbound alert mail against the OpenSRS relay. Without this committed a
fresh checkout cannot render /etc/msmtprc.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
esp32-stm32-vcu's scripts/release.sh uses jq to rewrite the protocol
manifest before publishing it to S3. The image did not have it, so the
release aborted at:
./scripts/release.sh: line 61: jq: command not found
The failure mode is nastier than a red job. jq is only reached *after*
the firmware .bin and version.json have already been uploaded, so
clients were served the new build while the git tag, the Gitea release
and the protocol manifest were never written — the repo still showed
the previous release as latest while production served a newer one.
Only jq was missing; aws and sha256sum are already present (verified in
the rebuilt image: jq-1.7, aws-ok, sha256sum-ok).
This surfaced when the skudak firmware jobs were moved into this image
(they previously ran on the default gitea-ci image, which has jq).
The import-only Snipcart key renders empty on normal deploys (import
endpoint 503s); it was injected via EXTRA_VARS just for the historical
refund re-import and is now blanked again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both are pulled on demand by pio run/buildfs; without them baked in, every
ephemeral job container re-downloads them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New opt-in job image localhost/gitea-ci-platformio:7.0.1 (pattern matches the
ESP-IDF image): PlatformIO core 6.1.19 with the espressif32@7.0.1 platform,
xtensa toolchain and Arduino framework pre-baked via a seed project so
ephemeral job containers download nothing. First consumer is
Skudak/esp32-web-interface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the ASCII-safe address transliteration fix (go-fulfillr#24) to prod
after dev verification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the ASCII-safe address transliteration fix (go-fulfillr#24) to the
staging back-office. Prod image tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns build to the prod back-office so the Coupons
admin round-trips the global-sale banner fields on prod. Applied via
`make deploy TAGS=fulfillr`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns bump (coupon banner_label/banner_text) to
the staging back-office so the Coupons admin round-trips the global-sale fields.
Applied via `make deploy TAGS=fulfillr-dev`. Prod tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.0.0 dropped pgvecto.rs support, which crash-looped immich-server
("No vector extension found") and left photos.debyl.io on a white page.
Swap db_image to Immich's bundled postgres image (VectorChord 0.4.3 +
pgvecto.rs 0.2.0) so v3 reindexes the existing embeddings into
VectorChord in place; bump server/ml from v2.7.5 to v3.0.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stripe relink endpoints (stripe-candidates / link-stripe) + ch_ refund support
now live on prod back-office.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.
- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
the alpine image), interval 30s / timeout 5s / retries 3 /
start_period 30s (covers the ~14s HMAC retry backoff), failure_action
restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
build carrying the /health/startup probe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The esp32-stm32-vcu firmware build generates common-yaml headers with
`python3 generate.py`, which needs pyyaml + jinja2. The runner's base Python is
PEP 668 externally-managed (pip install fails) and the IDF venv isn't on PATH in
the docker-exec step shell, so install both as distro packages. Lets firmware
jobs run a plain `python3 generate.py` with no pip and no IDF sourcing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds email_sent/email_failed timeline events for ticket label emails.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ticket refund + replacement-shipment endpoints and guarded transitions.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- remove snipcart_api_key from dev/production config (Snipcart decommissioned
post-migration)
- add review-outreach and cart-recovery schedule_name/schedule_group blocks
(dev + prod) for the EventBridge-driven outreach and cart-recovery jobs
- bump fulfillr image 20260607.0217 -> 20260613.0117
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Containerfile.ci: add python3-yaml + python3-jinja2 and the
gcc-arm-none-eabi / binutils / libnewlib toolchain for embedded builds
- bind-mount the runner's SSH key + known_hosts read-only into each job
container at /root/.ssh so submodule clones over
ssh://git@git.skudak.com:2222 succeed; staged as a dedicated
container_file_t-labelled ci-ssh copy (tasks/user.yml) and allowlisted
via valid_volumes (config.yaml.j2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ollama role and SearXNG container backed FISTO AI responses in the
greg-time Discord bot. greg-time 3.9.6 drops both (plus the Gemini path)
in favor of a single xAI Grok backend, so:
- remove the ollama role and its wiring in deploy_home.yml
- remove the searxng container task, template, and searxng_path default
- gregtime: swap OLLAMA_*/SEARXNG_URL/GEMINI_API_KEY env for XAI_API_KEY,
bump image 3.6.5 -> 3.9.6
- vault: add xai_api_key, drop gemini_api_key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The EasyPost tracker webhook moved to debyltech-api (publicly reachable Lambda);
the fulfillr host is LAN-restricted and no longer hosts it, so the carve-out is
no longer needed. Removes the handle blocks for prod and dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Fulfillr host is IP-restricted, so EasyPost's servers can't reach it. Add a
narrow `handle /webhooks/easypost` before the IP restriction (handle blocks are
mutually exclusive, first match wins) for prod (:9054) and dev (:9055) so the
HMAC-verified tracker webhook is reachable while the rest of the host stays locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lambda packaging steps in some workflows shell out to `zip`; the image
only had `unzip`. Add `zip` alongside it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a second go-fulfillr container (fulfillr-dev) wired to the staging
Turso store + EasyPost/Stripe test keys via dev.json, served at
fulfillr-dev.debyltech.com (Caddy -> :9055), LAN-restricted like prod.
- fulfillr-dev.yml + dev.json.j2: the staging container, volumes, config
- defaults: fulfillr_dev_* vars; prod store URL stubbed off until cutover
- Caddyfile + caddy.yml: fulfillr-dev site block and static mount
- awsddns.yml: Route53 DDNS for the fulfillr-dev hostname
- production.json.j2: add store_database_url/store_auth, rename stripe key
var to fulfillr_stripe_api_key
- vault.yml: dev + store/stripe secrets
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch the act_runners from :host execution to docker:// images backed by
a rootless podman socket under the gitea-runner user, so each job runs in
its own ephemeral container with per-job Go caches. This eliminates the
cross-repo GOMODCACHE/go-build poisoning that forced the debyl runner to
capacity:1.
- deps.yml: enable the rootless --user podman.socket, ensure subuid/subgid,
register gitea_runner_uid; drop the rootful system socket override,
podman-docker and host golang
- images.yml + Containerfile.ci/.espidf: build localhost/gitea-ci and
localhost/gitea-ci-espidf into the runner's rootless image store
- config.yaml.j2: docker:// labels (per-runner overridable), docker_host
-> rootless socket, force_pull false
- act_runner.service.j2: XDG_RUNTIME_DIR + DOCKER_HOST -> user socket
- defaults: uniform capacity:4 (drop the debyl capacity:1 workaround);
esp_idf_version now tags the espressif/idf-based image
- main.yml: import images.yml, drop the host esp-idf install (firmware jobs
use the espressif/idf job container instead)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump fulfillr image to the build with the tickets feature, and add the
tickets_table to the fulfillr production.json config (new debyltech-tickets-prod
DynamoDB table) so the /api/v1/tickets routes register.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- add stripe_api_key to fulfillr production.json template
- add restricted Stripe key to ansible vault (encrypted)
- bump fulfillr image to the CI build containing the Stripe endpoints
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up the case-status simplification from go-fulfillr 309550d
(only "open" and "closed" are accepted on PATCH; "new" is rejected).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Bump fulfillr container image from 20260124.0411 to 20260509.1940
(built from go-fulfillr commit 48b9f60 which adds /api/v1/cases
endpoints for the contact-form CRM dashboard).
- Add fulfillr_cases_table default ("debyltech-cases-prod") so the
HasCasesConfig() guard flips on at startup and the cases routes
register.
- Add cases_table to production.json.j2 so it lands in /config inside
the container.
Verified after deploy: GET /api/v1/cases returns the existing test
cases, PATCH succeeds, GSI1PK rewrite works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tools directory can exist without the Python venv being created,
causing install.sh to be skipped on re-runs. Check for python_env
instead, which is the actual output we need.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Generate ed25519 deploy key and add git.skudak.com/git.debyl.io host
keys to known_hosts so the runner can clone SSH submodules in CI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gitea-actions role now uses a `gitea_runners` list instead of a
single `gitea_instance_url`. Each instance gets its own config, systemd
service, working directory, and cache. Migrates from the old single
`act_runner.service` to per-instance `act_runner-{name}.service`.
Adds git.skudak.com alongside git.debyl.io as runner targets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add n8n container (n8nio/n8n:2.11.3) with Caddy reverse proxy at n8n.debyl.io
- Add --exclude .ssh to cloud backup rsync to prevent overwriting
authorized_keys on TrueNAS backup targets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SSH keys moved to /etc/ssh/backup_keys/ (ssh_home_t) and backup scripts
to /usr/local/bin/ (bin_t) to fix SELinux denials - container_file_t
context blocked rsync from exec'ing ssh. Also fixes skudak key path
mismatch (was truenas_skudak, key deployed as truenas_skudak-cloud).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces old 168-mod collection (3636931465) with new 385-mod collection.
Cleaned BBCode artifacts from mod IDs, updated map folders for 32 maps.
LogCabin retained for player connect/disconnect logging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ollama role for local LLM inference (install, service, models)
- Add searxng container for private search
- Migrate hostname from home.bdebyl.net to home.debyl.io
(inventory, awsddns, zomboid entrypoint, home_server_name)
- Update vault with new secrets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmarked uncensored models for the gregtime FISTO bot. dolphin-mistral
produces the best uncensored creative content, dolphin-phi is faster fallback.
Added OLLAMA_NUM_PREDICT env var (300) and bumped image to 3.3.0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move hardcoded image versions to variables defined in main.yml for
easier version management in one place.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move fluent-bit to common role (systemd service, not a container)
- Move geoip to podman/tasks/data/ (data prep, not a container)
- Remove debyltech tag from geoip (not a debyltech service)
- Fix check_mode for fetch subuid task to enable dry-run mode
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add uptime-kuma-personal container on port 3002
- Add Caddy config for uptime.debyl.io with IP restriction
- Update both uptime-kuma instances to 2.0.2
- Rename debyltech tag from uptime-kuma to uptime-debyltech
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update street2 address to Unit 95
- Add outreach config with DynamoDB tables and SES settings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename b42revamp server from "zomboidb42revamp" to "gregboid"
- Remove mod 3238830225 from workshop items
- Replace Real Firearms with B42RainsFirearmsAndGunPartsExpanded4213
- Remove 2788256295/ammomaker mod
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Deploy systemd path unit that watches for trigger file from Discord
bot and executes world reset script to delete saves and restart server.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Deploy systemd timer that writes zomboid container stats to
zomboid-stats.json every 30 seconds for gregtime to read.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update Gitea Skudak to use skudaknoreply_mail_* vault variables
- Remove redundant gitea_skudak_smtp_* variables from vault
- Update skudaknoreply credentials to noreply@skudak.com
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Gitea Skudak (git.skudak.com):
- New Gitea instance with PostgreSQL in podman pod under git user
- SSH access via Gitea's built-in SSH server on port 2222
- Registration restricted to @skudak.com emails with email confirmation
- SMTP configured for email delivery
Domain migrations:
- wiki.skudakrennsport.com → wiki.skudak.com (302 redirect)
- cloud.skudakrennsport.com + cloud.skudak.com (dual-domain serving)
- BookStack APP_URL updated to wiki.skudak.com
- Nextcloud trusted_domains updated for cloud.skudak.com
Infrastructure:
- SELinux context for git user container storage (container_file_t)
- Firewall rule for port 2222/tcp (Gitea Skudak SSH)
- Caddy reverse proxy for git.skudak.com
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change image source from AWS ECR to git.debyl.io/debyltech/fulfillr
- Update login task from ECR to Gitea registry authentication
- Add Gitea registry credentials to vault
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create gitea-runner user with podman access
- Install podman-docker for docker CLI compatibility
- Download and configure act_runner binary
- Systemd service for act_runner daemon
- Host-mode runner labels for Fedora
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Gitea container registry login task
- Add graylog.yml with full stack (MongoDB, OpenSearch, Graylog, gelf-proxy)
- Use container image instead of binary for gelf-proxy
- Image tagged from git.debyl.io/debyltech/gelf-proxy
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
