Bastian de Byl
0ab423ca55
The data-only rsync left no way to restore a working instance: mysql/ and config/ were never backed up, so a recovery would have files but no shares, users or metadata. Dump the database before syncing files (a DB older than the files is repairable with occ files:scan; a newer one references blobs that never made it into the backup) and ship config/ alongside it. Capture the --chmod=Du=rwx,Dgo=rx flag that had been hand-added to the deployed skudak-cloud script. It was outside git, so every deploy silently reverted it. It now lives in backup_rsync_extra_args. Add OnFailure= alerting. The units failed silently before, which is how an iDrive sync failure sat unnoticed since May. msmtp rather than the esmtp already installed: the OpenSRS relay is port 465 (implicit TLS) and libesmtp only speaks STARTTLS. Exclude nextcloud.log* from the sync and cap log_rotate_size. skudak-cloud was running at loglevel 0 and had written a 64 GB log that was being rsynced and pushed to S3; set it to 2 to match the home instance. Stagger the timers (04:00 / 04:30) so both finish before the 05:00 TrueNAS snapshot task, and bound TimeoutStartSec so a wedged rsync cannot leave the unit activating forever and skip every subsequent trigger. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
41 lines
1.6 KiB
Django/Jinja
41 lines
1.6 KiB
Django/Jinja
#!/bin/bash
|
|
# {{ ansible_managed }}
|
|
# OnFailure= handler for the Nextcloud backup units. Invoked as:
|
|
# nextcloud-backup-alert.sh <failed-unit-name>
|
|
#
|
|
# Deliberately NOT `set -e`: an alert handler that dies partway through
|
|
# reports nothing, which is worse than a partial report. Same reasoning as
|
|
# roles/ups/templates/ups-restore.sh.j2.
|
|
set -uo pipefail
|
|
|
|
TAG=nextcloud-backup
|
|
UNIT="${1:-unknown}"
|
|
TO="{{ backup_alert_email | default('root') }}"
|
|
HOST="$(hostname -f 2>/dev/null || hostname)"
|
|
|
|
result="$(systemctl show -p Result --value "$UNIT" 2>/dev/null)"
|
|
code="$(systemctl show -p ExecMainStatus --value "$UNIT" 2>/dev/null)"
|
|
|
|
# One machine-parseable line for Graylog, then the context.
|
|
logger -t "$TAG" -p daemon.err -- \
|
|
"status=failed unit=$UNIT result=${result:-unknown} exit=${code:-unknown}"
|
|
|
|
body="$(printf 'Nextcloud backup FAILED on %s\n\nunit: %s\nresult: %s\nexit: %s\n\n--- last 40 journal lines ---\n' \
|
|
"$HOST" "$UNIT" "${result:-unknown}" "${code:-unknown}")
|
|
$(journalctl -u "$UNIT" -n 40 --no-pager -o cat 2>/dev/null)"
|
|
|
|
echo "$body" | logger -t "$TAG" -p daemon.err
|
|
|
|
# Mail is best-effort: if the MTA is not configured the journald record above
|
|
# is still the authoritative signal, so never fail the handler on this.
|
|
if command -v sendmail >/dev/null 2>&1; then
|
|
printf 'To: %s\nSubject: [%s] Nextcloud backup FAILED: %s\nContent-Type: text/plain; charset=UTF-8\n\n%s\n' \
|
|
"$TO" "$HOST" "$UNIT" "$body" | sendmail -t \
|
|
&& logger -t "$TAG" -p daemon.info -- "alert_mail=sent to=$TO" \
|
|
|| logger -t "$TAG" -p daemon.err -- "alert_mail=failed to=$TO"
|
|
else
|
|
logger -t "$TAG" -p daemon.err -- "alert_mail=skipped reason=no-sendmail"
|
|
fi
|
|
|
|
exit 0
|