A de Byl Technologies LLC Nextcloud cloned from the Skudak instance:
LibreSign signing for people without an account, registration off
(admin-created accounts only), no Group Folders. DNS is a terraform-managed
ALIAS to fulfillr.debyltech.com.
- containers/debyltech/cloud.yml: nextcloud/mariadb/redis on port 8091.
It installs unattended on the first deploy, sends mail through SES as
noreply@debyltech.com, and re-asserts the Skudak LibreSign settings.
- files/debyltechmail: skudakmail rebranded, with a new black-and-white
wordmark and white web-UI logos.
- LibreSign is pinned to 14.2.2 from the GitHub release (sha256-checked)
rather than `occ app:install`. The app store served a same-day 14.2.3
whose tarball has no binary-signature metadata. 14.2.x also doesn't
create its own download dirs, so they're pre-created.
- The backup runs nightly at 04:15 to TrueNAS /mnt/glacier/debyltechcloud and
reaches personal iDrive via the "iDrive E2 Backup" task; the TrueNAS side
excludes /debyltechcloud/_backup/config/**.
- Fix the libresign:configure:check gate in both instances: '\berror\b'
becomes a backspace in Jinja and never matched, so a check reporting three
errors passed clean. Now '\\berror\\b'.
- vault: cloud_debyltech_* secrets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sunset automation required TV mode off, so an afternoon of TV skipped
the driveway string lights entirely (Sep 22 and 23) - not an outage. The
driveway now has its own sunset -1h automation, with catch-up on restart
or switch reconnect before 23:00.
Evening brightness lives in one script (evening_lights_apply) that blends
between the old step levels; a 5-minute ramp from 20:30 eases lights that
are on and leaves alone any a person has changed by hand. The Dining Hall
no longer bumps to 50% at 21:30.
TV off after 23:30 now only turns off the living room glow instead of
bringing the whole house back to full brightness, and TV on/off leave the
lights alone in daylight. Lights-out moves to 23:30, and a 01:00 sweep
catches anything switched back on at the wall.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daily quote keeps a ledger of what it has posted and re-rolls ZenQuotes
until it finds something the channel has not read, with the header framing and
the offline fallback pool drawn against that same ledger.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runner's job images were built by ansible into localhost/ only, so the
nightly CI prune deleted them and every idle stretch ended with CI failing in
under a second on `docker pull localhost/gitea-ci:latest` until someone re-ran
the role and waited out a rebuild. The previous commit moved them to the Gitea
registry; this moves the *build* off the deploy path entirely.
- .gitea/workflows/ci-images.yml builds files/Containerfile.* and pushes to
git.debyl.io/gitbot/. Per-image change detection, so an ESP-IDF pin bump does
not rebuild the other two; weekly schedule for base-image updates; a
workflow_dispatch selector. PRs build under a throwaway :pr-<n> tag and drop
it -- the build lands in the live runner's store, and act_runner will not
re-pull a tag it already has, so a PR using the real tag would hand every
later job on this host an unmerged image.
- The Containerfiles stop being ansible templates: their version vars are now
--build-arg, read by the workflow out of the same defaults/main.yml the role
interpolates, so CI and ansible build the same bytes from one set of pins.
- LABEL io.debyl.ci-base moves into each Containerfile so neither builder can
forget the prune exemption; the workflow re-checks it before pushing.
- roles/gitea-actions pulls instead of building. gitea_ci_build_local=true
restores the local build+push for seeding a cold registry or when CI is
down -- the workflow that builds gitea-ci runs in gitea-ci.
- Lint .gitea/ alongside ansible/, and document the flow in the role README.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nights are checkboxes again: a household ticks every night that works, which
also says which nights do not. At least one tick (or "Any of these works")
is required.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Compacts the admin invite table so it fits without clipping: short token link
with Copy/Msg, status as an emoji, Qty, allergens/notes behind popovers, and
relative update times.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes the admin invite table clipping its Edit/Revoke column, and adds
default-headcount people estimates to the invited/awaiting summary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the anonymized "Who's coming" card for guests who have answered, and a
message-template copy button on the admin invite table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A single Go binary with SQLite, built and loaded as localhost/rsvpd:<VERSION>
by make deploy-remote in ~/src/rsvp-debylio. Both of its listeners are
published on 127.0.0.1 only: Caddy proxies the public one to everyone and the
admin one (/admin, no login) only to caddy_local_networks. A Caddyfile mistake
alone cannot expose admin, and neither can a port mistake alone.
Guests' invite links are the credential and they sit in the URL path, which
shapes the vhost:
- It does not import common_headers. That snippet sets Referrer-Policy
same-origin, which would replace the app's no-referrer and let a token leak
in a Referer header.
- Its access log rewrites request>uri to /i/REDACTED and drops the Location
response header, since every POST 303s back to /i/<token>.
- Caddy's error logger is separate from the site's and wrote the raw URI to
caddy.log when the upstream was down. The global log now excludes
http.log.error.rsvp and a filtered rsvp-errors logger takes it instead.
Verified with zero token occurrences in both logs, locally and live.
The data directory is owned directly by the host uid of the container's uid
10001 (subuid + 10000). Setting it to the podman user and chowning back each run
flipped ownership on every deploy and briefly locked the app out of its
database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rolling a world back meant hand-work over ssh: stop the service, move the live
save aside, unzip the right archive, chown into the container's subuid range,
relabel, start. That is the wrong shape of task to do by hand, and it is always
done under time pressure -- by construction, because the archive you want is
being deleted while you work.
PZ keeps BackupsCount=10 per set and writes one every BackupsPeriod=30 minutes,
so a periodic backup is reachable for about five hours and then gone. On
2026-09-05 the snapshot the admins asked for (05:16, four minutes before the
incident) had about 90 minutes of life left when the request came in.
Same shape as the wipe: the Discord bot writes a trigger file into its own rw
volume, zomboid-restore.path notices it, and zomboid-restore.service runs the
script as the podman user. The bot gets no ssh, no systemd, and keeps only its
existing read-only mount of the Zomboid volume.
Two details carry most of the correctness.
Resolution is by mtime, not by index. The rotation renames the files -- today's
backup_7.zip is backup_8.zip half an hour from now, and a new backup_7.zip holds
a different world -- so an index is valid only while the listing is fresh, which
is not long enough to survive a human reading a confirmation prompt. The trigger
names a set and an mtime; the script resolves the path itself, whitelists the
filename, and refuses if nothing matches. It never accepts a path.
Everything that can fail is checked before the server is touched. A rotated-out
target, an archive with no debbzoid world in it, a bad action, a traversal
attempt in the set name: each aborts with the server still running and writes a
result file the bot reports back. The live world is moved aside rather than
deleted, so a restore is undoable and the last three are kept.
One thing PZ does not advertise: its backups do not cover the whole save
directory. blam/, a mod's own state, is in none of them -- not the 05:16 archive
and not the newest one. Restoring only what the archive holds therefore lands
the world slightly *behind* the target rather than on it, so anything present in
the displaced world and absent from the archive is carried across.
The gregtime tag moves to 3.17.0 for the bot half of this -- `backups`,
`restore <n>`, `restore confirm`, `restore undo`, gated to the same two admins
as the wipe. That image is built and running on the host; its source is not
committed yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016QdWYhwUtwM2NQGukiRh12
assistant.debyl.io returned 400 on every request. With use_x_forwarded_for
set, home assistant hard-fails any request carrying X-Forwarded-For from an
address outside trusted_proxies:
Received X-Forwarded-For header from an untrusted proxy 169.254.2.1
caddy runs --network host and proxies to localhost:8123, so the source address
is whatever pasta presents inside the container's netns. Rootless podman
switched from slirp4netns to pasta in v5 (this host runs 5.7.1), which moved
that address from 10.0.2.x -- covered by the existing 10.0.0.0/8 entry -- to a
link-local tap0 address that nothing in the list matched. The container has no
10.x address at all any more.
Latent since the pasta migration; it only surfaced when the site was next
opened. Reproduced directly:
curl -H 'X-Forwarded-For: 1.2.3.4' http://localhost:8123/ -> 400
Bumped to 2026.8.3 in the same pass (identical digest to the stable tag).
The config directory was snapshotted first: the 2026.5.1 -> 2026.8.3 database
migration is not reversible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Build 41 mod (versionMin=41.60) carried into Sophie's B42 list. It indexes
the ModOptions framework at VehicleDoorsHotkey_Options.lua:4, that framework is
not in the list, and so it dies there every boot:
attempted index: ModOptions of non-table: null
Twenty-one exceptions a boot for a hotkey that has never once worked. Excluded
rather than fixed by adding ModOptions, since nothing else in the list wants it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reset script died on its very first line. log() piped through tee into
zomboid/logs/world-reset.log, that directory was owned by the container's
subuid, and the script runs as the podman user -- so tee returned EACCES and
set -e killed the run before it stopped the server or touched a save. Every
`@bot` reset since had been a no-op that reported nothing.
Nothing mounts zomboid/logs into a container; it only holds output from
host-side helpers running as the podman user, so it is now owned by that user
rather than by the subuid the container volumes need.
log() no longer treats the file as load-bearing either. stdout is already
captured by the journal, so an unwritable log is worth continuing past rather
than aborting a wipe over.
Verified end to end by writing the trigger exactly as the bot does: server
stopped, saves and player database deleted, service restarted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The anti-scanner rules dropped 53-byte Steam query packets above 1/hour with a
burst of 2, per source IP, on the theory that only scanners send queries and
that real players would already have been marked verified by the priority-2
rule when they sent something larger.
That premise is inverted. A client's first contact with the server IS a 53-byte
query, so nobody can be verified before querying, and nobody can query more
than twice an hour without being dropped. The counters said so plainly: five
packets had ever matched the verified-accept rule, against 30,563 drops.
fail2ban then banned each dropped player for a week -- 32 live bans, 550 total,
firing every ten minutes, every one of them a residential address.
Keeps the shape of the protection and moves the threshold somewhere no real
client reaches: opening the server browser or retrying a connection is a
handful of queries, a flood is thousands. Removes the old rules first so hosts
carrying them converge instead of stacking a second copy.
The bans themselves were also hooked into INPUT for tcp only, so they never
blocked the UDP game traffic they were meant to -- the drop rule was doing all
the damage on its own. Cleared the outstanding 32.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud follows the personal instance, which is now verified on 34.0.3:
`occ upgrade` completed, needsDbUpgrade is false, and cloud.debyl.io reports
34.0.3.2 out of maintenance mode.
BookStack skips 26.4 -- 26.3.4 is simply where this instance sat. Note the tag
is 26.5.4, not 26.05.4: solidnerd drops the month's leading zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Immich moves the server and machine-learning images only; the Postgres and
Redis images stay on their pinned digests, which are tied to the vectorchord
extension version rather than the app release. v3.1.0 is a quality-of-life
release -- its one breaking change is dropping iOS 14 in the mobile client,
which does not touch the server.
Nextcloud 34.0.2 -> 34.0.3 is a patch release; the official image's entrypoint
runs `occ upgrade` itself on first start with a newer version.
Skudak's Nextcloud is deliberately left on 34.0.2 until this one is verified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The logrotate config added with the Sophie overhaul claimed to bound PZ's own
logs. It did not. data/Logs had reached 18 GB across 112 entries and 146k
files, and rotation had never once fired there.
Two reasons, both wrong assumptions on my part. PZ rolls its logs into dated
directories -- logs_2025-12-14/ through logs_2026-08-26/, up to 279 MB each --
so the Logs/*.txt glob only ever matched a handful of loose files at the top.
And it rotates by size: not one file in that tree exceeds 100M, because the
growth is in the number of files, not the size of any of them.
Age-based pruning is the right tool, so this adds a daily zomboid-log-prune
timer keeping 14 days and deleting the emptied directories behind it. It runs
under podman unshare, since the files belong to the container's UID.
logrotate keeps server-console.txt, which is a single ever-growing file and
genuinely is what it is good at. Narrowed its scope to say so rather than
implying coverage it never had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deployment had drifted badly. It was pinned to the Steam `-beta unstable`
branch, which stopped being right the moment B42 became the default at 42.20,
and it wrote every mod ID with the `\` prefix that B42 required only during
that unstable period and now rejects. Three server profiles were carried
around (vanilla, modded, b42revamp) whose mod lists were hand-curated blobs
from a Steam collection that has since moved on.
Moves to the Sophie 42 community preset, vendored from GerDeathstar/sophie-pz
("Sophie 42 Files.zip", 2026-07-31 == the b42 release tag): 283 mod IDs, 256
workshop items, map_distanciado over Muldraugh, plus its SandboxVars,
spawnregions and spawnpoints. Sophie's gameplay settings are kept exactly as
shipped -- PVP with its damage modifiers, the safety system, MaxPlayers=32,
PauseEmpty, no sleep, safehouses off. Only the keys this deployment actually
owns are templated over the top.
The modlist is stored as YAML lists rather than the INI's semicolon blobs, so
the next Sophie update produces a diff you can read. Exclusions live in
zomboid_mods_excluded with a reason each, seeded with IconsInventory -- already
absent upstream, listed so a re-vendor cannot quietly bring it back.
Config is now seeded before first boot instead of patched after it. The old
approach could not write the INI until the server had generated one, so every
setting went through lineinfile guarded on a stat; seeding the whole file from
the preset removes the chicken-and-egg and puts the config in git. force is
off by design: these files are a starting point, not managed state, so the
server can be stopped, hand-edited and regenerated without Ansible clobbering
the edits. Push them again deliberately with -e zomboid_config_force=true.
The B41 Discord keys were dead. B42 replaced DiscordChannel/DiscordChannelID
with DiscordChatChannel, which takes a channel name rather than a snowflake,
so the DiscordChannelID=... this role had been appending was a key the server
ignores and the chat bridge has not been working.
Renaming the server to debbzoid is what starts the new world: PZ derives the
INI, save directory and player DB from the name, so gregboid stays on disk
untouched as a rollback.
Re-enabled, because the reason it was off is now fixed. It was disabled for
saturating the SSD -- ~10 MB/s of log writes into the shared journal, which
starved Gitea CI badly enough to stretch a firmware build to 17 minutes. The
container now logs to its own rotating k8s-file instead of the journal every
other service shares, and logrotate caps the server's own server-console.txt
and Logs/ at 100M keeping 3. copytruncate is mandatory there: PZ holds those
fds for its whole life, so a rename would leave it writing into an unlinked
inode. Memory is unchanged; idle draw is nowhere near MAX_RAM.
Also stops the entrypoint recursively chowning the install tree on every boot.
With 256 workshop mods that is a large inode walk and pure churn once the
first run has set ownership.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Graylog was the worst cost/benefit tenant on this 4-core box: two JVMs plus
MongoDB holding ~1.6 GB resident and ~3% CPU around the clock to store ~3k
messages a day -- about 28 MB across its four live indices. journald already
retains ~25 days of the same logs at its 500M cap, so this costs searchability,
not the logs.
The switch is `graylog_enabled` in inventory rather than a role default,
because three roles read it (common, podman, graylog-config). The disabled
path is an active teardown, not a skipped create: the containers already on
the host keep running and their systemd user units keep restarting them at
boot unless something stops and removes them. fluent-bit follows the same
flag -- with the GELF sink down it would spin retrying a dead 127.0.0.1:12202
and fill the journal it exists to drain -- but only the service state follows,
so re-enabling is a restart rather than a reinstall.
Caddy reloads were silently no-ops. The handler read /etc/caddy/Caddyfile,
which is a single-file bind mount, and podman binds those by inode; the
template module writes a temp file and renames it into place, so every deploy
gave the host file a new inode while the container kept seeing the one it was
created with. Config changes only ever landed when something recreated the
container. {{ caddy_path }}/config is also mounted, as a *directory*, and
directory mounts resolve names at open() time -- so /config/Caddyfile is
always the file Ansible just wrote.
awsddns and its four siblings had accumulated 12 zombies over 30 days of
uptime. The image's PID 1 is busybox crond, which only waitpid()s the job PIDs
it tracks and does no generic orphan reaping, so whenever the run-parts/sh
layer exited before the script it left a permanent <defunct>. init: true puts
catatonit at PID 1 to reap them, and the recreation clears the existing ones.
Also bumps fulfillr and greg-time-bot images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TrueNAS was power-cycled, the CIFS mounts failed, and systemd never retried
-- mount units are not restarted on failure. SMB came back, nothing
remounted, and immich-server served an empty library for days while its
database still listed 14,181 assets pointing at /mnt/media/originals.
fstab carried no _netdev, no nofail and no automount, so there was no path
back without a human. Now:
x-systemd.automount any access re-attempts the mount; failure stops
being terminal
_netdev / nofail ordered after network-online, dead NAS cannot block boot
soft I/O errors instead of blocking forever, so the
container can be restarted rather than wedging in
uninterruptible sleep
idle-timeout unmount when unused, clearing stale handles
resilienthandles SMB3 rides out brief blips
ansible.posix.mount mounts directly and never starts the generated
.automount unit, leaving the on-access trigger inactive -- enable it
explicitly, or the headline fix silently does nothing.
The containers are systemd USER units while the mounts are SYSTEM units, so
RequiresMountsFor= is unavailable. cifs-watchdog bridges the scopes: checks
health, recovers, and restarts ONLY immich-server (the sole consumer of both
paths; postgres/redis/ML use local volumes).
Two bugs the umount test caught, both worth knowing:
- `ls` cannot test mountedness. An unmounted mount point is an ordinary
empty directory, so ls succeeds and recovery was skipped entirely.
- A drop repaired within a single run leaves prev=healthy, so keying the
restart solely on the stored state skipped it while the container still
held its stale view.
Also moves the SMB password out of /etc/fstab, which is 0644 and was
readable by every local user, into a 0600 credentials file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LibreSign had been silently broken since it was first deployed in
January. Every step of the old before-starting hook ended in `|| echo`,
so six months of failures logged nothing.
LibreSign repair
- Root cause was a stale config_path: a valid OpenSSL root CA existed at
generation 1, a failed CFSSL attempt left an empty generation 2, and
config_path was left pointing at the empty one. Regenerated as
"Skudak LLP" (was the pre-rename "Skudak Rennsport LLP").
- Deleted the hook. Java/PDFtk/jSignPdf live under data/appdata_*, a
persisted volume, so they only ever needed installing once. Install and
verification are now explicit tasks that actually fail.
- PHP_MEMORY_LIMIT 1024M -- the 512M image default fails opaquely
mid-signature. LC_ALL/LANG so the JVM is not ANSI_X3.4-1968.
- signature_render_mode=GRAPHIC_ONLY. Any other mode halves the stamp
width and overlays a name/date block that collides with the drawn mark
and duplicates what our documents already typeset. The value must be
exactly GRAPHIC_ONLY; a bare "GRAPHIC" is accepted by occ, matches no
radio in the UI, and silently reverts to default.
- write_qrcode_on_footer=false, written with --type=boolean because
FooterHandler reads it via getValueBool and the typed appconfig API
does not coerce a string "0". The validation URL text is kept.
- identification_documents=0 -- the default gates signing behind an ID
upload plus admin approval, so signers saw no way to sign.
- shareapi_restrict_user_enumeration_full_match=no, so an email owned by
an existing account can be added as a signer. Root cause is in core
(MailPlugin.php:163), not LibreSign. Do NOT set full_match_email=no --
that disables email signer search entirely.
Mail branding (skudakmail app)
- Two supported extension points, no core patch and no LibreSign fork:
mail_template_class for layout, subjects, button labels and the footer
LibreSign never adds; and a BeforeMessageSent listener to embed the
wordmark as a cid: part so it survives remote-image blocking.
- A third listener adds scoped CSS fixing the signing page being clipped
on iOS Safari (100vh -> 100dvh). Patched upstream too.
- skudakmail-verify.php.j2 asserts all of the above through the real
useTemplate() path and fails the play on drift. Every assertion was
proven to fail when deliberately regressed.
Redis
- memcache.locking was unset, so Nextcloud used DBLockingProvider and
every file lock became a MariaDB write -- the contention behind the
intermittent multi-second stalls. Verified after: db locks static,
redis keys growing.
- requirepass lives in a mounted 0640 conf, not --requirepass, which
would leak it into podman inspect, the systemd unit and ps. The file is
chowned to uid 999 because redis-server does not run as root and the
:ro mount stops the image fixing it itself.
- No maxmemory: cache is evictable, locks are NOT, and evicting a held
lock permits concurrent writers to one file. No persistence either --
a restored RDB could reinstate locks whose owner is long dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not
holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across
16 log streams. Made explicit at 10MiB/keep 3/7d.
Note roll_size et al are subdirectives of `output file`, NOT of `log`.
Getting that wrong does not degrade gracefully: Caddy refuses to start on a
bad config, so every site went down until it was corrected. Worth a
`caddy validate` gate before reload.
journald had no SystemMaxUse and had reached 4 GB, drifting toward its
10%-of-filesystem default (~190 GB on this root). Capped at 500M.
Both are safe to keep short because fluent-bit ships the journal and every
Caddy access log into Graylog -- though note its GELF output has been
erroring for days, which weakens that premise and wants investigating.
The larger find was unrelated to logs: 896 images totalling 59.6 GB with
75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and
5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees
orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added
a weekly timer keeping 30 days so a rollback still needs no rebuild.
Also dropped the decommissioned 6379/tcp redis rule (nothing listening;
Immich's redis is on the shared podman network) and the orphaned nosql, s3
and searxng volume dirs.
Backup log exclusions turned out to be unnecessary: Gitea logs to console
so its log dirs are empty, Nextcloud already excludes its own, BookStack
mounts only uploads, and Caddy is not backed up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs
both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min,
host at 10% charge) and best-effort IPMI power-on when mains returns. The 10%
threshold leans on ignorelb + override.battery.charge.low rather than a custom
poller, because CyberPower asserts its own low-battery flag far too early.
Credentials come from vault vars; nothing sensitive is templated in the clear.
Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which
expects an external caller every ~5 minutes, and nothing was calling. The
personal instance had not run a background job since 2026-05-14 and skudak since
2024-11-20. Consequently trash and file versions never expired, stale chunked
uploads accumulated, calendar reminders never fired, and nextcloud.log was never
rotated -- which quietly made the existing log_rotate_size cap inert. Added a
systemd timer per instance, skipping cleanly when the container is down or in
maintenance so deploy windows don't show up as failed units.
Trash retention on the personal instance: the default "auto" only expires when
disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB
free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional
while still purging early under pressure.
Image bumps:
nextcloud 33.0.0 -> 34.0.2 (both cloud and skudak-cloud)
greg-time-bot 3.9.25 -> 3.10.0
fulfillr 20260723.2044 -> 20260728.2155 (prod and dev)
The fulfillr bump records what is already deployed: both containers were rolled
to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer
update campaign). Committing it keeps the repo from claiming an older tag than
the host is actually running, which would otherwise roll fulfillr backwards on
the next clean-checkout deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The import-only Snipcart key renders empty on normal deploys (import
endpoint 503s); it was injected via EXTRA_VARS just for the historical
refund re-import and is now blanked again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the ASCII-safe address transliteration fix (go-fulfillr#24) to prod
after dev verification.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the ASCII-safe address transliteration fix (go-fulfillr#24) to the
staging back-office. Prod image tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns build to the prod back-office so the Coupons
admin round-trips the global-sale banner fields on prod. Applied via
`make deploy TAGS=fulfillr`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploys the go-store banner-columns bump (coupon banner_label/banner_text) to
the staging back-office so the Coupons admin round-trips the global-sale fields.
Applied via `make deploy TAGS=fulfillr-dev`. Prod tag unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.0.0 dropped pgvecto.rs support, which crash-looped immich-server
("No vector extension found") and left photos.debyl.io on a white page.
Swap db_image to Immich's bundled postgres image (VectorChord 0.4.3 +
pgvecto.rs 0.2.0) so v3 reindexes the existing embeddings into
VectorChord in place; bump server/ml from v2.7.5 to v3.0.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stripe relink endpoints (stripe-candidates / link-stripe) + ch_ refund support
now live on prod back-office.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.
- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
the alpine image), interval 30s / timeout 5s / retries 3 /
start_period 30s (covers the ~14s HMAC retry backoff), failure_action
restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
build carrying the /health/startup probe).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds email_sent/email_failed timeline events for ticket label emails.
Deployed to fulfillr-dev and fulfillr (prod) on home.debyl.io.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
