Bastian de Byl
bc110ce69e
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
2.9 KiB
Django/Jinja
70 lines
2.9 KiB
Django/Jinja
#!/bin/bash
|
|
# {{ ansible_managed }}
|
|
# OnFailure= handler for every backup unit (Nextcloud and Gitea). Invoked as:
|
|
# nextcloud-backup-alert.sh <failed-unit-name>
|
|
#
|
|
# One shared copy serves all instances -- cloud-backup.yml renders it once and
|
|
# later includes are no-ops -- so the wording here must not name one product.
|
|
# The failing UNIT name is what identifies the instance.
|
|
#
|
|
# Deliberately NOT `set -e`: an alert handler that dies partway through
|
|
# reports nothing, which is worse than a partial report. Same reasoning as
|
|
# roles/ups/templates/ups-restore.sh.j2.
|
|
set -uo pipefail
|
|
|
|
TAG=nextcloud-backup
|
|
UNIT="${1:-unknown}"
|
|
TO="{{ backup_alert_email | default('root') }}"
|
|
HOST="$(hostname -f 2>/dev/null || hostname)"
|
|
|
|
result="$(systemctl show -p Result --value "$UNIT" 2>/dev/null)"
|
|
code="$(systemctl show -p ExecMainStatus --value "$UNIT" 2>/dev/null)"
|
|
|
|
# Only claim a failure when the unit actually reports one. Starting this
|
|
# handler by hand (or any other spurious trigger) would otherwise mail out a
|
|
# subject line saying FAILED about a run that succeeded. Keeping status=failed
|
|
# exact also stops such triggers matching the Graylog alert rule.
|
|
if [ "${result:-success}" = "success" ]; then
|
|
state=spurious
|
|
prio=daemon.warning
|
|
headline="$(printf 'Backup alert handler was invoked on %s, but %s reports SUCCESS.\nThis is not a backup failure -- most likely the handler was started manually.' "$HOST" "$UNIT")"
|
|
subject="[$HOST] Backup alert (spurious, unit OK): $UNIT"
|
|
else
|
|
state=failed
|
|
prio=daemon.err
|
|
headline="Backup FAILED on $HOST"
|
|
subject="[$HOST] Backup FAILED: $UNIT"
|
|
fi
|
|
|
|
# One machine-parseable line for Graylog, then the context.
|
|
logger -t "$TAG" -p "$prio" -- \
|
|
"status=$state unit=$UNIT result=${result:-unknown} exit=${code:-unknown}"
|
|
|
|
body="$(printf '%s\n\nunit: %s\nresult: %s\nexit: %s\n\n--- last 40 journal lines ---\n' \
|
|
"$headline" "$UNIT" "${result:-unknown}" "${code:-unknown}")
|
|
$(journalctl -u "$UNIT" -n 40 --no-pager -o cat 2>/dev/null)"
|
|
|
|
echo "$body" | logger -t "$TAG" -p "$prio"
|
|
|
|
# Only genuine failures are worth an email. A spurious invocation carries no
|
|
# action for a human, and mailing it trains the reader to ignore the subject
|
|
# line -- which defeats the point of having the alert at all. The journald
|
|
# record above is kept either way, so spurious triggers stay greppable.
|
|
if [ "$state" != "failed" ]; then
|
|
logger -t "$TAG" -p daemon.info -- "alert_mail=skipped reason=$state"
|
|
exit 0
|
|
fi
|
|
|
|
# Mail is best-effort: if the MTA is not configured the journald record above
|
|
# is still the authoritative signal, so never fail the handler on this.
|
|
if command -v sendmail >/dev/null 2>&1; then
|
|
printf 'To: %s\nSubject: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n' \
|
|
"$TO" "$subject" "$body" | sendmail -t \
|
|
&& logger -t "$TAG" -p daemon.info -- "alert_mail=sent to=$TO" \
|
|
|| logger -t "$TAG" -p daemon.err -- "alert_mail=failed to=$TO"
|
|
else
|
|
logger -t "$TAG" -p daemon.err -- "alert_mail=skipped reason=no-sendmail"
|
|
fi
|
|
|
|
exit 0
|