A spurious invocation carries no action for a reader, so mailing it just
trains them to skip past the subject line -- which defeats the point of the
alert. Send mail only when the unit actually reports failure; spurious
triggers still leave their journald record, so they stay greppable and can
still feed a Graylog rule.
Verified both paths: a spurious trigger leaves /var/log/msmtp.log untouched
and logs alert_mail=skipped reason=spurious, while a genuinely failed unit
still composes mail with the FAILED subject.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mariadb-upgrade --force has now been run against both instances and repaired
the system tables, so --routines and --events no longer abort the dump.
Restore them for completeness. Both verified: rc=0 with a clean completion
trailer, and both Nextcloud instances report installed with unchanged table
counts afterwards.
The upgrade still exits non-zero on these containers because it cannot create
the `sys` schema: /var/lib/mysql is owned by daemon rather than mysql, so
mysqld may not create top-level databases. `sys` is diagnostic only and
unused by Nextcloud, but the same permission would block creating any new
database, so it is recorded in the template comment.
The alert handler claimed FAILED in its subject line regardless of what the
unit actually reported, so starting it by hand mailed out a failure notice
for a run that succeeded. Derive the subject and log line from the real
Result, and emit status=spurious rather than status=failed so such triggers
cannot match a Graylog alert rule keyed on genuine failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-only rsync left no way to restore a working instance: mysql/ and
config/ were never backed up, so a recovery would have files but no shares,
users or metadata. Dump the database before syncing files (a DB older than
the files is repairable with occ files:scan; a newer one references blobs
that never made it into the backup) and ship config/ alongside it.
Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the
deployed skudak-cloud script. It was outside git, so every deploy silently
reverted it. It now lives in backup_rsync_extra_args.
Add OnFailure= alerting. The units failed silently before, which is how an
iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp
already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp
only speaks STARTTLS.
Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud
was running at loglevel 0 and had written a 64 GB log that was being rsynced
and pushed to S3; set it to 2 to match the home instance.
Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS
snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the
unit activating forever and skip every subsequent trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add n8n container (n8nio/n8n:2.11.3) with Caddy reverse proxy at n8n.debyl.io
- Add --exclude .ssh to cloud backup rsync to prevent overwriting
authorized_keys on TrueNAS backup targets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces old 168-mod collection (3636931465) with new 385-mod collection.
Cleaned BBCode artifacts from mod IDs, updated map folders for 32 maps.
LogCabin retained for player connect/disconnect logging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bastian de Byl