Backup hardening, UPS monitoring, Nextcloud cron, image bumps #10

2026-07-30T13:05:12-04:00

bastian commented

2026-07-30 13:05:12 -04:00

Six commits: the backup-hardening work plus UPS monitoring, Nextcloud background jobs, and container image bumps.

Backup hardening

Nextcloud backups now take real DB dumps, with alerting and a drift fix
home_smtp vault key added so the host can send system mail
Full dump flags restored; stopped spurious alerts that claimed FAILED when nothing had
Alerting narrowed to genuine failures only, so the mail is worth reading

ups role

NUT server on home.debyl.io for the CyberPower PR1500RT2U backing both it and truenas.localdomain. Staged shutdown — TrueNAS sheds ~200 W at t+2min via its own native shutdown timer, host goes down at 10% charge — plus best-effort IPMI power-on when mains returns.

The 10% threshold uses ignorelb + override.battery.charge.low rather than a custom poller, because CyberPower asserts its own low-battery flag around 20–35%, far too early to act on. Credentials come from vault vars; nothing sensitive is templated in the clear.

Nextcloud background jobs

Both instances run backgroundjobs_mode: cron, which expects an external caller every ~5 minutes — and nothing was calling. The personal instance hadn't run a background job since 2026-05-14, skudak since 2024-11-20.

That meant trash and file versions never expired, stale chunked uploads accumulated, calendar reminders never fired, and nextcloud.log was never rotated — which quietly made the existing log_rotate_size cap inert.

A systemd timer per instance now drives cron.php. It skips cleanly when the container is absent or in maintenance, so deploys and image bumps don't parade as failed units, and Type=oneshot gets an explicit TimeoutStartSec so a wedged run can't sit in "activating" forever and silently swallow every later tick.

Trash retention

Nextcloud's default auto only expires trash under disk pressure, so 66 GB of >30-day deletions sat on a host with 1.3 TB free — effectively unbounded. auto, 30 makes the 30-day expiry unconditional while still purging early when space is short.

Image bumps

	from	to
nextcloud (cloud + skudak-cloud)	33.0.0	34.0.2
greg-time-bot	3.9.25	3.10.0
fulfillr (prod + dev)	20260723.2044	20260728.2155

The fulfillr line records what is already running — both containers were rolled to that image on 2026-07-29 for SCRUM-156. Without it the repo claims an older tag than the host has, and the next make deploy TAGS=fulfillr from a clean checkout would roll fulfillr backwards, breaking the portal against the new UI.

Checks

make lint (yamllint) clean
All new/changed YAML parses
roles/ups/ is committed alongside deploy_home.yml deliberately — the playbook now declares - role: ups, so splitting them would leave it referencing a role absent from the repo
ansible/vars/vault.yml verified $ANSIBLE_VAULT;1.1;AES256 on disk and stored git-crypt-encrypted in the object store, not plaintext

🤖 Generated with Claude Code

Six commits: the backup-hardening work plus UPS monitoring, Nextcloud background jobs, and container image bumps. ## Backup hardening - Nextcloud backups now take real DB dumps, with alerting and a drift fix - `home_smtp` vault key added so the host can send system mail - Full dump flags restored; stopped spurious alerts that claimed FAILED when nothing had - Alerting narrowed to genuine failures only, so the mail is worth reading ## ups role NUT server on `home.debyl.io` for the CyberPower PR1500RT2U backing both it and `truenas.localdomain`. Staged shutdown — TrueNAS sheds ~200 W at t+2min via its own native shutdown timer, host goes down at 10% charge — plus best-effort IPMI power-on when mains returns. The 10% threshold uses `ignorelb` + `override.battery.charge.low` rather than a custom poller, because CyberPower asserts its own low-battery flag around 20–35%, far too early to act on. Credentials come from vault vars; nothing sensitive is templated in the clear. ## Nextcloud background jobs Both instances run `backgroundjobs_mode: cron`, which expects an external caller every ~5 minutes — and nothing was calling. The personal instance hadn't run a background job since **2026-05-14**, skudak since **2024-11-20**. That meant trash and file versions never expired, stale chunked uploads accumulated, calendar reminders never fired, and `nextcloud.log` was never rotated — which quietly made the existing `log_rotate_size` cap inert. A systemd timer per instance now drives `cron.php`. It skips cleanly when the container is absent or in maintenance, so deploys and image bumps don't parade as failed units, and `Type=oneshot` gets an explicit `TimeoutStartSec` so a wedged run can't sit in "activating" forever and silently swallow every later tick. ## Trash retention Nextcloud's default `auto` only expires trash under disk pressure, so 66 GB of >30-day deletions sat on a host with 1.3 TB free — effectively unbounded. `auto, 30` makes the 30-day expiry unconditional while still purging early when space is short. ## Image bumps | | from | to | |---|---|---| | nextcloud (cloud + skudak-cloud) | 33.0.0 | 34.0.2 | | greg-time-bot | 3.9.25 | 3.10.0 | | fulfillr (prod + dev) | 20260723.2044 | 20260728.2155 | The fulfillr line **records what is already running** — both containers were rolled to that image on 2026-07-29 for SCRUM-156. Without it the repo claims an older tag than the host has, and the next `make deploy TAGS=fulfillr` from a clean checkout would roll fulfillr *backwards*, breaking the portal against the new UI. ## Checks - `make lint` (yamllint) clean - All new/changed YAML parses - `roles/ups/` is committed alongside `deploy_home.yml` deliberately — the playbook now declares `- role: ups`, so splitting them would leave it referencing a role absent from the repo - `ansible/vars/vault.yml` verified `$ANSIBLE_VAULT;1.1;AES256` on disk and stored git-crypt-encrypted in the object store, not plaintext 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bastian added 6 commits 2026-07-30 13:05:13 -04:00

feat(gitea-actions): install jq in the ESP-IDF CI image 1e1d53ecd8

esp32-stm32-vcu's scripts/release.sh uses jq to rewrite the protocol
manifest before publishing it to S3. The image did not have it, so the
release aborted at:

    ./scripts/release.sh: line 61: jq: command not found

The failure mode is nastier than a red job. jq is only reached *after*
the firmware .bin and version.json have already been uploaded, so
clients were served the new build while the git tag, the Gitea release
and the protocol manifest were never written — the repo still showed
the previous release as latest while production served a newer one.

Only jq was missing; aws and sha256sum are already present (verified in
the rebuilt image: jq-1.7, aws-ok, sha256sum-ok).

This surfaced when the skudak firmware jobs were moved into this image
(they previously ran on the default gitea-ci image, which has jq).

harden nextcloud backups: db dumps, alerting, drift fix 0ab423ca55

The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.

Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.

Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.

Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.

Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

add home_smtp vault key for system mail 7c72aee4f9

Referenced by roles/common/templates/msmtp/msmtprc.j2 to authenticate
outbound alert mail against the OpenSRS relay. Without this committed a
fresh checkout cannot render /etc/msmtprc.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

restore full dump flags; stop spurious alerts claiming FAILED 6e99794d0f

mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.

The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.

The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

only email genuine backup failures 16145fb6bd

A spurious invocation carries no action for a reader, so mailing it just
trains them to skip past the subject line -- which defeats the point of the
alert. Send mail only when the unit actually reports failure; spurious
triggers still leave their journald record, so they stay greppable and can
still feed a Graylog rule.

Verified both paths: a spurious trigger leaves /var/log/msmtp.log untouched
and logs alert_mail=skipped reason=spurious, while a genuinely failed unit
still composes mail with the FAILED subject.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

UPS monitoring, Nextcloud cron, container image bumps 5776dbe1bf

ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs
both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min,
host at 10% charge) and best-effort IPMI power-on when mains returns. The 10%
threshold leans on ignorelb + override.battery.charge.low rather than a custom
poller, because CyberPower asserts its own low-battery flag far too early.
Credentials come from vault vars; nothing sensitive is templated in the clear.

Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which
expects an external caller every ~5 minutes, and nothing was calling. The
personal instance had not run a background job since 2026-05-14 and skudak since
2024-11-20. Consequently trash and file versions never expired, stale chunked
uploads accumulated, calendar reminders never fired, and nextcloud.log was never
rotated -- which quietly made the existing log_rotate_size cap inert. Added a
systemd timer per instance, skipping cleanly when the container is down or in
maintenance so deploy windows don't show up as failed units.

Trash retention on the personal instance: the default "auto" only expires when
disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB
free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional
while still purging early under pressure.

Image bumps:
  nextcloud       33.0.0  -> 34.0.2   (both cloud and skudak-cloud)
  greg-time-bot   3.9.25  -> 3.10.0
  fulfillr        20260723.2044 -> 20260728.2155  (prod and dev)

The fulfillr bump records what is already deployed: both containers were rolled
to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer
update campaign). Committing it keeps the repo from claiming an older tag than
the host is actually running, which would otherwise roll fulfillr backwards on
the next clean-checkout deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bastian merged commit a63bf5edec into master

2026-07-30 13:05:20 -04:00

bastian deleted branch backup-hardening

2026-07-30 13:05:20 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bastian/deploy_home#10