back up Gitea + Skudak app data; drop PartKeepr and Pi-hole
Extends the Nextcloud backup machinery rather than adding a second
mechanism. cloud-backup.sh.j2 gains three guarded options, all no-ops for
the existing callers:
backup_podman_user Gitea runs rootless under `git`, not `podman`
backup_db_type postgres (Gitea) and mysql (BookStack) alongside
mariadb; each engine's completion trailer differs,
and grepping for the wrong one fails every run
backup_sqlite_dbs `sqlite3 .backup` for live WAL-mode SQLite, gated on
`pragma integrity_check` before promotion -- rsync
is either stale (no -wal) or torn (with it)
New instances: gitea-debyl, skudak-gitea, bookstack, partsy-skudak. The
alert handler is rendered once and shared, so its wording is now generic
rather than per-product; TAG stays nextcloud-backup because an external
Graylog rule matches on it.
`apply:` on the includes is load-bearing -- tags on a dynamic
include_tasks do not reach the tasks inside it.
Business data (skudak-gitea, bookstack, partsy-skudak) goes to TrueNAS
and on to Skudak's own iDrive account; the personal bucket's
/skudak*/** excludes are permanent, not a stopgap.
Removals: PartKeepr is superseded by Partsy, and its teardown never
finished -- it targeted /etc/systemd/system/podman-partkeepr*.service,
wrong prefix and wrong scope, leaving enabled user units in failed state.
Pi-hole's role was already orphaned (absent from deploy_home.yml); its
port 53 rule went with it after confirming nothing listens there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +0,0 @@
|
|||||||
---
|
|
||||||
deps: [
|
|
||||||
php-sqlite,
|
|
||||||
php-fpm
|
|
||||||
]
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
---
|
|
||||||
dependencies:
|
|
||||||
- role: http
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
---
|
|
||||||
- name: install pi-hole-server
|
|
||||||
ansible.builtin.command: yay -S --noconfirm pi-hole-server
|
|
||||||
args:
|
|
||||||
creates: /bin/pihole
|
|
||||||
|
|
||||||
- name: install pi-hole-server dependencies
|
|
||||||
become: true
|
|
||||||
ansible.builtin.package:
|
|
||||||
name: "{{ deps }}"
|
|
||||||
state: present
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
---
|
|
||||||
- import_tasks: deps.yml
|
|
||||||
- import_tasks: php.yml
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
---
|
|
||||||
- name: replace pi.hole hostname
|
|
||||||
become: true
|
|
||||||
ansible.builtin.replace:
|
|
||||||
path: "{{ item }}"
|
|
||||||
regexp: "pi\\.hole"
|
|
||||||
replace: "pi.bdebyl.net"
|
|
||||||
loop:
|
|
||||||
- /srv/http/pihole/admin/scripts/pi-hole/php/auth.php
|
|
||||||
- /srv/http/pihole/pihole/index.php
|
|
||||||
tags:
|
|
||||||
- pihole
|
|
||||||
@@ -24,7 +24,6 @@ gregtime_path: "{{ podman_volumes }}/gregtime"
|
|||||||
hass_path: "{{ podman_volumes }}/hass"
|
hass_path: "{{ podman_volumes }}/hass"
|
||||||
# nginx_path: removed - nginx no longer used
|
# nginx_path: removed - nginx no longer used
|
||||||
# nosql_path: removed - nosql/redis no longer used
|
# nosql_path: removed - nosql/redis no longer used
|
||||||
partkeepr_path: "{{ podman_volumes }}/partkeepr"
|
|
||||||
partsy_path: "{{ podman_volumes }}/partsy"
|
partsy_path: "{{ podman_volumes }}/partsy"
|
||||||
partsy_skudak_path: "{{ podman_volumes }}/partsy-skudak"
|
partsy_skudak_path: "{{ podman_volumes }}/partsy-skudak"
|
||||||
photos_path: "{{ podman_volumes }}/photos"
|
photos_path: "{{ podman_volumes }}/photos"
|
||||||
@@ -75,7 +74,6 @@ zomboid_maps:
|
|||||||
modded: "Muldraugh, KY"
|
modded: "Muldraugh, KY"
|
||||||
b42revamp: "AnruisiTown;Asakusa lake town;BananaBeachHouse;Blackstone;Cathaya Valley2.0;Cathaya Valley2.0 highway;Daisy County;DawnTown;Estate 39;Fort Waterfront B42;Greenleaf;GreenRiver;IrisEyot;Louisville_Quarantine_Zone;Louisville_River_Marina;Louisville_Riverboat;Maplewood;Meiya'sTown;Muldraugh_FireDept;muldraughmilitarybaseas24;PeaceTown;RaccoonCity;Raven Creek B42;map_distanciado;rvupdate;rv2;serenitybunker;Tikitown;Uncle Red's Bunker B42;Uncle Red's Bunker Redux B42;WestPoint-MilitaryTugBoat;Muldraugh, KY"
|
b42revamp: "AnruisiTown;Asakusa lake town;BananaBeachHouse;Blackstone;Cathaya Valley2.0;Cathaya Valley2.0 highway;Daisy County;DawnTown;Estate 39;Fort Waterfront B42;Greenleaf;GreenRiver;IrisEyot;Louisville_Quarantine_Zone;Louisville_River_Marina;Louisville_Riverboat;Maplewood;Meiya'sTown;Muldraugh_FireDept;muldraughmilitarybaseas24;PeaceTown;RaccoonCity;Raven Creek B42;map_distanciado;rvupdate;rv2;serenitybunker;Tikitown;Uncle Red's Bunker B42;Uncle Red's Bunker Redux B42;WestPoint-MilitaryTugBoat;Muldraugh, KY"
|
||||||
|
|
||||||
pihole_path: "{{ podman_volumes }}/pihole"
|
|
||||||
sshpass_cron_path: "{{ podman_volumes }}/sshpass_cron"
|
sshpass_cron_path: "{{ podman_volumes }}/sshpass_cron"
|
||||||
caddy_path: "{{ podman_volumes }}/caddy"
|
caddy_path: "{{ podman_volumes }}/caddy"
|
||||||
|
|
||||||
|
|||||||
@@ -1,59 +0,0 @@
|
|||||||
---
|
|
||||||
# PartKeepr has been replaced by Partsy
|
|
||||||
# This playbook removes PartKeepr containers and services
|
|
||||||
# Keeping MySQL data volume for historical reference only
|
|
||||||
|
|
||||||
- name: stop and remove partkeepr container
|
|
||||||
become: true
|
|
||||||
become_user: "{{ podman_user }}"
|
|
||||||
containers.podman.podman_container:
|
|
||||||
name: partkeepr
|
|
||||||
state: absent
|
|
||||||
|
|
||||||
- name: stop and remove partkeepr-db container
|
|
||||||
become: true
|
|
||||||
become_user: "{{ podman_user }}"
|
|
||||||
containers.podman.podman_container:
|
|
||||||
name: partkeepr-db
|
|
||||||
state: absent
|
|
||||||
|
|
||||||
- name: remove systemd service for partkeepr
|
|
||||||
become: true
|
|
||||||
ansible.builtin.systemd:
|
|
||||||
name: "podman-partkeepr.service"
|
|
||||||
state: stopped
|
|
||||||
enabled: false
|
|
||||||
daemon_reload: true
|
|
||||||
ignore_errors: true
|
|
||||||
|
|
||||||
- name: remove systemd service for partkeepr-db
|
|
||||||
become: true
|
|
||||||
ansible.builtin.systemd:
|
|
||||||
name: "podman-partkeepr-db.service"
|
|
||||||
state: stopped
|
|
||||||
enabled: false
|
|
||||||
daemon_reload: true
|
|
||||||
ignore_errors: true
|
|
||||||
|
|
||||||
- name: remove systemd service files for partkeepr
|
|
||||||
become: true
|
|
||||||
ansible.builtin.file:
|
|
||||||
path: "{{ item }}"
|
|
||||||
state: absent
|
|
||||||
loop:
|
|
||||||
- "/etc/systemd/system/podman-partkeepr.service"
|
|
||||||
- "/etc/systemd/system/podman-partkeepr-db.service"
|
|
||||||
notify: systemd daemon-reload
|
|
||||||
|
|
||||||
- name: preserve partkeepr mysql data volume for history
|
|
||||||
become: true
|
|
||||||
ansible.builtin.file:
|
|
||||||
path: "{{ partkeepr_path }}/mysql"
|
|
||||||
state: directory
|
|
||||||
owner: "{{ podman_subuid.stdout }}"
|
|
||||||
group: "{{ podman_user }}"
|
|
||||||
mode: 0755
|
|
||||||
notify: restorecon podman
|
|
||||||
|
|
||||||
- name: flush handlers
|
|
||||||
ansible.builtin.meta: flush_handlers
|
|
||||||
@@ -161,6 +161,32 @@
|
|||||||
cron_container: skudak-cloud
|
cron_container: skudak-cloud
|
||||||
cron_script_path: /usr/local/bin/skudak-cloud-cron.sh
|
cron_script_path: /usr/local/bin/skudak-cloud-cron.sh
|
||||||
|
|
||||||
|
# This instance is BUSINESS data and deliberately backs up to TrueNAS ONLY.
|
||||||
|
#
|
||||||
|
# It used to reach personal cloud storage too: the TrueNAS "iDrive E2 Backup"
|
||||||
|
# cloud-sync task pushes /mnt/glacier to a personal iDrive e2 bucket, which
|
||||||
|
# swept skudakcloud/ along with it. That task now carries an explicit
|
||||||
|
# `/skudakcloud/**` exclude, and on 2026-07-30 the stranded copy was purged
|
||||||
|
# from the bucket -- business data does not belong in personal storage.
|
||||||
|
#
|
||||||
|
# The copy was also worthless as a backup: 30 objects against 20,802 files on
|
||||||
|
# TrueNAS (0.14%), stale since 2026-05-20. Worse, the bucket is VERSIONED and
|
||||||
|
# the sync runs in COPY mode (never deletes), so every daily run retained
|
||||||
|
# another ~60 GB version of the pre-cap nextcloud.log -- 56 of them, 3.46 TB,
|
||||||
|
# 99.3% of a 3.49 TB footprint. Deleting current objects alone reclaims
|
||||||
|
# nothing on a versioned bucket; the versions must be purged explicitly.
|
||||||
|
#
|
||||||
|
# Offsite is now BUSINESS-OWNED: Skudak's own iDrive e2 account, bucket
|
||||||
|
# `backup-all`, pushed by TrueNAS cloud-sync task "Skudak iDrive - Nextcloud"
|
||||||
|
# (id 8, /mnt/glacier/skudakcloud -> /skudakcloud, daily 06:00). That bucket
|
||||||
|
# has a 90-day NoncurrentVersionExpiration policy so the version bloat above
|
||||||
|
# cannot repeat. The personal task's `/skudakcloud/**` exclude is PERMANENT --
|
||||||
|
# it is what keeps business data out of personal storage, not a stopgap.
|
||||||
|
#
|
||||||
|
# Still outstanding: the data itself lives on personal TrueNAS hardware. To
|
||||||
|
# finish separating, add an S3 stage to cloud-backup.sh.j2 guarded by a
|
||||||
|
# `backup_s3_*` var so only this instance opts in -- awscli2 is already
|
||||||
|
# installed on the host -- and then drop the TrueNAS rsync below.
|
||||||
- include_tasks: containers/cloud-backup.yml
|
- include_tasks: containers/cloud-backup.yml
|
||||||
vars:
|
vars:
|
||||||
backup_name: skudak-cloud
|
backup_name: skudak-cloud
|
||||||
|
|||||||
@@ -15,12 +15,9 @@
|
|||||||
- 443/tcp
|
- 443/tcp
|
||||||
# Gitea Skudak SSH
|
# Gitea Skudak SSH
|
||||||
- 2222/tcp
|
- 2222/tcp
|
||||||
# pihole (unused?)
|
|
||||||
- 53/tcp
|
|
||||||
- 53/udp
|
|
||||||
# nosql/redis
|
# nosql/redis
|
||||||
- 6379/tcp
|
- 6379/tcp
|
||||||
# ???
|
# BookStack (wiki.skudak.com) -- container publishes 6875:8080
|
||||||
- 6875/tcp
|
- 6875/tcp
|
||||||
# Satisfactory
|
# Satisfactory
|
||||||
- 7777/tcp
|
- 7777/tcp
|
||||||
|
|||||||
@@ -34,12 +34,6 @@
|
|||||||
image: ghcr.io/home-assistant/home-assistant:2026.5.1
|
image: ghcr.io/home-assistant/home-assistant:2026.5.1
|
||||||
tags: hass
|
tags: hass
|
||||||
|
|
||||||
- import_tasks: containers/home/partkeepr.yml
|
|
||||||
vars:
|
|
||||||
db_image: docker.io/library/mariadb:10.0
|
|
||||||
image: docker.io/bdebyl/partkeepr:0.1.10
|
|
||||||
tags: partkeepr
|
|
||||||
|
|
||||||
- import_tasks: containers/home/partsy.yml
|
- import_tasks: containers/home/partsy.yml
|
||||||
vars:
|
vars:
|
||||||
image: "git.debyl.io/debyltech/partsy:latest"
|
image: "git.debyl.io/debyltech/partsy:latest"
|
||||||
@@ -117,3 +111,126 @@
|
|||||||
image: docker.io/cm2network/steamcmd:root
|
image: docker.io/cm2network/steamcmd:root
|
||||||
tags: zomboid
|
tags: zomboid
|
||||||
|
|
||||||
|
# ---------------------------------------------------------- Gitea backups
|
||||||
|
# The Gitea pods themselves are owned by roles/git, but the backup machinery
|
||||||
|
# (containers/cloud-backup.yml plus templates/nextcloud/*) lives here, and an
|
||||||
|
# include_tasks reaching across roles would need a path outside the role. So
|
||||||
|
# the two Gitea backup instances are wired here alongside the container ones.
|
||||||
|
#
|
||||||
|
# Both run as ROOTLESS podman under "{{ git_user }}", not "{{ podman_user }}"
|
||||||
|
# -- hence backup_podman_user -- and use PostgreSQL rather than MariaDB.
|
||||||
|
#
|
||||||
|
# Scheduled ahead of the 04:00/04:30 Nextcloud runs so everything lands before
|
||||||
|
# the 05:00 TrueNAS snapshot.
|
||||||
|
# `apply` is required: tags on a dynamic include_tasks select whether the
|
||||||
|
# include runs, but do NOT propagate to the tasks inside it, so without this
|
||||||
|
# `make deploy TAGS=gitea-backup` includes the file and then filters out every
|
||||||
|
# task in it. The Nextcloud instances avoid this only because they are reached
|
||||||
|
# through a static import_tasks chain that tags their children at parse time.
|
||||||
|
- include_tasks:
|
||||||
|
file: containers/cloud-backup.yml
|
||||||
|
apply:
|
||||||
|
tags: gitea-backup
|
||||||
|
vars:
|
||||||
|
backup_name: gitea-debyl
|
||||||
|
backup_product: Gitea
|
||||||
|
backup_podman_user: "{{ git_user }}"
|
||||||
|
data_path: "{{ git_home }}/volumes/gitea/data"
|
||||||
|
db_container: gitea-debyl-postgres
|
||||||
|
backup_db_type: postgres
|
||||||
|
ssh_key_path: /etc/ssh/backup_keys/gitea
|
||||||
|
ssh_key_content: "{{ gitea_backup_ssh_key }}"
|
||||||
|
ssh_user: gitea
|
||||||
|
remote_path: /mnt/glacier/gitea
|
||||||
|
script_path: /usr/local/bin/gitea-backup.sh
|
||||||
|
# actions_log/artifacts are CI churn (510 MB and growing) and rebuildable;
|
||||||
|
# indexers/queues/tmp are derived state Gitea recreates on start. Note the
|
||||||
|
# default excludes are Nextcloud-specific, so this must be set explicitly.
|
||||||
|
backup_rsync_excludes: >-
|
||||||
|
--exclude '/gitea/actions_log' --exclude '/gitea/actions_artifacts'
|
||||||
|
--exclude '/gitea/tmp' --exclude '/gitea/indexers' --exclude '/gitea/queues'
|
||||||
|
backup_oncalendar: "*-*-* 03:30:00"
|
||||||
|
tags: gitea-backup
|
||||||
|
|
||||||
|
# ------------------------------------------------- Skudak app-data backups
|
||||||
|
# BookStack (wiki.skudak.com) and partsy-skudak are BUSINESS data. Both share
|
||||||
|
# one TrueNAS dataset (/mnt/glacier/skudakapps) so they need only one backup
|
||||||
|
# user, key and cloud-sync task between them; the personal "iDrive E2 Backup"
|
||||||
|
# task excludes /skudakapps/** and Skudak's own task pushes it to backup-all.
|
||||||
|
#
|
||||||
|
# Scheduled ahead of the 03:30+ Gitea/Nextcloud jobs and the 05:00 snapshot.
|
||||||
|
- include_tasks:
|
||||||
|
file: containers/cloud-backup.yml
|
||||||
|
apply:
|
||||||
|
tags: [skudak, skudak-apps-backup]
|
||||||
|
vars:
|
||||||
|
backup_name: bookstack
|
||||||
|
backup_product: BookStack
|
||||||
|
data_path: "{{ bookstack_path }}"
|
||||||
|
db_container: bookstack-db
|
||||||
|
# mysql:5.7 predates the mariadb-dump alias -- see cloud-backup.sh.j2.
|
||||||
|
backup_db_type: mysql
|
||||||
|
ssh_key_path: /etc/ssh/backup_keys/skudakapps
|
||||||
|
ssh_key_content: "{{ skudakapps_backup_ssh_key }}"
|
||||||
|
ssh_user: skudakapps
|
||||||
|
remote_path: /mnt/glacier/skudakapps/bookstack
|
||||||
|
script_path: /usr/local/bin/bookstack-backup.sh
|
||||||
|
# The wiki content is the DATABASE; /mysql is its raw datadir, which must
|
||||||
|
# not be rsynced live -- the dump above is the consistent copy. public/
|
||||||
|
# and storage/ hold the uploads and are the only file trees worth shipping.
|
||||||
|
backup_rsync_excludes: "--exclude '/mysql'"
|
||||||
|
backup_oncalendar: "*-*-* 03:00:00"
|
||||||
|
tags: skudak, skudak-apps-backup
|
||||||
|
|
||||||
|
- include_tasks:
|
||||||
|
file: containers/cloud-backup.yml
|
||||||
|
apply:
|
||||||
|
tags: [skudak, skudak-apps-backup]
|
||||||
|
vars:
|
||||||
|
backup_name: partsy-skudak
|
||||||
|
backup_product: Partsy
|
||||||
|
data_path: "{{ partsy_skudak_path }}"
|
||||||
|
# Live WAL-mode SQLite: snapshotted via `sqlite3 .backup` rather than
|
||||||
|
# rsynced, so the -wal/-shm sidecars are deliberately excluded from the
|
||||||
|
# file tree -- shipping them alongside a separately-taken snapshot would
|
||||||
|
# only invite a confusing restore.
|
||||||
|
backup_sqlite_dbs:
|
||||||
|
- "{{ partsy_skudak_path }}/data/partsy.db"
|
||||||
|
backup_rsync_excludes: "--exclude '*-wal' --exclude '*-shm'"
|
||||||
|
ssh_key_path: /etc/ssh/backup_keys/skudakapps
|
||||||
|
ssh_key_content: "{{ skudakapps_backup_ssh_key }}"
|
||||||
|
ssh_user: skudakapps
|
||||||
|
remote_path: /mnt/glacier/skudakapps/partsy-skudak
|
||||||
|
script_path: /usr/local/bin/partsy-skudak-backup.sh
|
||||||
|
backup_oncalendar: "*-*-* 03:10:00"
|
||||||
|
tags: skudak, skudak-apps-backup
|
||||||
|
|
||||||
|
# BUSINESS data. Rsynced to TrueNAS here, then pushed offsite to SKUDAK'S OWN
|
||||||
|
# iDrive e2 account (bucket `backup-all`) by TrueNAS cloud-sync task "Skudak
|
||||||
|
# iDrive - Gitea" (id 9, /mnt/glacier/skudakgit -> /skudakgit, daily 06:30).
|
||||||
|
#
|
||||||
|
# The personal "iDrive E2 Backup" task's /skudakgit/** exclude is PERMANENT:
|
||||||
|
# it is what keeps business data out of personal storage. Do not remove it --
|
||||||
|
# Skudak has its own task and bucket instead.
|
||||||
|
- include_tasks:
|
||||||
|
file: containers/cloud-backup.yml
|
||||||
|
apply:
|
||||||
|
tags: [skudak, gitea-backup-skudak]
|
||||||
|
vars:
|
||||||
|
backup_name: skudak-gitea
|
||||||
|
backup_product: Gitea
|
||||||
|
backup_podman_user: "{{ git_user }}"
|
||||||
|
data_path: "{{ git_home }}/volumes/gitea-skudak/data"
|
||||||
|
db_container: gitea-skudak-postgres
|
||||||
|
backup_db_type: postgres
|
||||||
|
ssh_key_path: /etc/ssh/backup_keys/skudak-gitea
|
||||||
|
ssh_key_content: "{{ skudakgit_backup_ssh_key }}"
|
||||||
|
ssh_user: skudakgit
|
||||||
|
remote_path: /mnt/glacier/skudakgit
|
||||||
|
script_path: /usr/local/bin/skudak-gitea-backup.sh
|
||||||
|
backup_rsync_excludes: >-
|
||||||
|
--exclude '/gitea/actions_log' --exclude '/gitea/actions_artifacts'
|
||||||
|
--exclude '/gitea/tmp' --exclude '/gitea/indexers' --exclude '/gitea/queues'
|
||||||
|
backup_oncalendar: "*-*-* 03:45:00"
|
||||||
|
tags: skudak, gitea-backup-skudak
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Nextcloud {{ instance_name }} backup to TrueNAS
|
Description={{ backup_product | default('Nextcloud') }} {{ instance_name }} backup to TrueNAS
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
OnFailure=nextcloud-backup-failed@%n.service
|
OnFailure=nextcloud-backup-failed@%n.service
|
||||||
|
|||||||
@@ -1,14 +1,17 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# {{ ansible_managed }}
|
# {{ ansible_managed }}
|
||||||
# Nextcloud "{{ backup_name }}" -> truenas.localdomain.
|
# {{ backup_product | default('Nextcloud') }} "{{ backup_name }}" -> truenas.localdomain.
|
||||||
#
|
#
|
||||||
# Ordering is deliberate: the database is dumped BEFORE the file tree is
|
# Ordering is deliberate: the database is dumped BEFORE the file tree is
|
||||||
# synced. A DB snapshot slightly OLDER than the files degrades to "files
|
# synced. A DB snapshot slightly OLDER than the files degrades to "files
|
||||||
# Nextcloud has not indexed yet" and is repaired with `occ files:scan`. A DB
|
# the app has not indexed yet" and is repaired with a rescan. A DB snapshot
|
||||||
# snapshot NEWER than the files references blobs that never made it into the
|
# NEWER than the files references blobs that never made it into the backup,
|
||||||
# backup, which surfaces as broken shares and dead file entries on restore.
|
# which surfaces as broken shares and dead file entries on restore.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Tag stays "nextcloud-backup" for EVERY instance, Gitea included: an external
|
||||||
|
# Graylog rule matches status=failed on this tag, and renaming it here would
|
||||||
|
# silently stop alerting for all of them. Misnomer retained deliberately.
|
||||||
TAG=nextcloud-backup
|
TAG=nextcloud-backup
|
||||||
INSTANCE={{ backup_name }}
|
INSTANCE={{ backup_name }}
|
||||||
STAGE={{ backup_stage_path | default('/var/backups/nextcloud/' ~ backup_name) }}
|
STAGE={{ backup_stage_path | default('/var/backups/nextcloud/' ~ backup_name) }}
|
||||||
@@ -26,9 +29,10 @@ log "status=start"
|
|||||||
|
|
||||||
{% if db_container | default('') %}
|
{% if db_container | default('') %}
|
||||||
# ------------------------------------------------------------ 1. database
|
# ------------------------------------------------------------ 1. database
|
||||||
# The Nextcloud containers are ROOTLESS podman owned by "{{ podman_user }}",
|
# The containers are ROOTLESS podman owned by
|
||||||
# but this script runs as root under systemd. Every podman call therefore
|
# "{{ backup_podman_user | default(podman_user) }}" -- Nextcloud runs under
|
||||||
# goes through sudo:
|
# {{ podman_user }}, Gitea under its own git user -- but this script runs as
|
||||||
|
# root under systemd. Every podman call therefore goes through sudo:
|
||||||
# -H HOME becomes the podman user's home, so podman finds its rootless
|
# -H HOME becomes the podman user's home, so podman finds its rootless
|
||||||
# graph root under ~/.local/share/containers
|
# graph root under ~/.local/share/containers
|
||||||
# cd; required preamble (see CLAUDE.md) so the shell starts in that home
|
# cd; required preamble (see CLAUDE.md) so the shell starts in that home
|
||||||
@@ -37,9 +41,12 @@ log "status=start"
|
|||||||
# guarded anyway so podman falls back cleanly if it ever does not.
|
# guarded anyway so podman falls back cleanly if it ever does not.
|
||||||
#
|
#
|
||||||
# No credential is stored in this file or placed on a host command line:
|
# No credential is stored in this file or placed on a host command line:
|
||||||
# $MYSQL_ROOT_PASSWORD and $MYSQL_DATABASE are expanded by the shell INSIDE
|
# $MYSQL_ROOT_PASSWORD / $MYSQL_DATABASE (or $POSTGRES_* for postgres) are
|
||||||
# the database container, which already carries them in its environment.
|
# expanded by the shell INSIDE the database container, which already carries
|
||||||
|
# them in its environment.
|
||||||
|
{% if backup_db_type | default('mariadb') != 'postgres' %}
|
||||||
#
|
#
|
||||||
|
# MariaDB-specific history, for the flag set below:
|
||||||
# --routines and --events initially aborted the dump here: mysql.proc read as
|
# --routines and --events initially aborted the dump here: mysql.proc read as
|
||||||
# corrupted (error 1728) and the event scheduler reported disabled (1577),
|
# corrupted (error 1728) and the event scheduler reported disabled (1577),
|
||||||
# both artefacts of an image bump without mariadb-upgrade. `mariadb-upgrade
|
# both artefacts of an image bump without mariadb-upgrade. `mariadb-upgrade
|
||||||
@@ -51,8 +58,14 @@ log "status=start"
|
|||||||
# by daemon rather than mysql, so mysqld may not create new top-level
|
# by daemon rather than mysql, so mysqld may not create new top-level
|
||||||
# databases. `sys` is purely diagnostic and unused by Nextcloud, so this is
|
# databases. `sys` is purely diagnostic and unused by Nextcloud, so this is
|
||||||
# cosmetic -- but it does mean creating a NEW database would fail too.
|
# cosmetic -- but it does mean creating a NEW database would fail too.
|
||||||
|
{% else %}
|
||||||
|
#
|
||||||
|
# --clean --if-exists makes the dump replayable into an existing database
|
||||||
|
# (`psql -U gitea gitea < dump.sql`) rather than only into an empty one, and
|
||||||
|
# --no-owner keeps it restorable when the target role names differ.
|
||||||
|
{% endif %}
|
||||||
pexec() {
|
pexec() {
|
||||||
sudo -H -u {{ podman_user }} bash -c \
|
sudo -H -u {{ backup_podman_user | default(podman_user) }} bash -c \
|
||||||
'cd; d=/run/user/$(id -u); [ -d "$d" ] && export XDG_RUNTIME_DIR="$d"
|
'cd; d=/run/user/$(id -u); [ -d "$d" ] && export XDG_RUNTIME_DIR="$d"
|
||||||
exec podman "$@"' _ "$@"
|
exec podman "$@"' _ "$@"
|
||||||
}
|
}
|
||||||
@@ -63,21 +76,40 @@ rm -f "$tmp"
|
|||||||
|
|
||||||
log "dumping {{ db_container }}"
|
log "dumping {{ db_container }}"
|
||||||
set +e
|
set +e
|
||||||
|
{% if backup_db_type | default('mariadb') == 'postgres' %}
|
||||||
|
pexec exec {{ db_container }} sh -c '
|
||||||
|
exec env PGPASSWORD="$POSTGRES_PASSWORD" pg_dump -U "$POSTGRES_USER" \
|
||||||
|
--no-owner --clean --if-exists "$POSTGRES_DB"
|
||||||
|
' | gzip -6 > "$tmp"
|
||||||
|
{% elif backup_db_type | default('mariadb') == 'mysql' %}
|
||||||
|
{# Upstream mysql:5.7 ships mysqldump, NOT the mariadb-dump alias that only
|
||||||
|
appears in MariaDB 10.5+. Flags and completion trailer are identical to the
|
||||||
|
MariaDB branch; 5.7.21 was verified to accept --no-tablespaces. #}
|
||||||
|
pexec exec {{ db_container }} sh -c '
|
||||||
|
exec env MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysqldump -u root \
|
||||||
|
--single-transaction --quick --routines --events --triggers \
|
||||||
|
--no-tablespaces --default-character-set=utf8mb4 "$MYSQL_DATABASE"
|
||||||
|
' | gzip -6 > "$tmp"
|
||||||
|
{% else %}
|
||||||
pexec exec {{ db_container }} sh -c '
|
pexec exec {{ db_container }} sh -c '
|
||||||
exec env MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb-dump -u root \
|
exec env MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb-dump -u root \
|
||||||
--single-transaction --quick --routines --events --triggers \
|
--single-transaction --quick --routines --events --triggers \
|
||||||
--no-tablespaces --default-character-set=utf8mb4 "$MYSQL_DATABASE"
|
--no-tablespaces --default-character-set=utf8mb4 "$MYSQL_DATABASE"
|
||||||
' | gzip -6 > "$tmp"
|
' | gzip -6 > "$tmp"
|
||||||
|
{% endif %}
|
||||||
dump_rc=${PIPESTATUS[0]}
|
dump_rc=${PIPESTATUS[0]}
|
||||||
set -e
|
set -e
|
||||||
[ "$dump_rc" -eq 0 ] || fail "mariadb-dump {{ db_container }} exited $dump_rc"
|
[ "$dump_rc" -eq 0 ] || fail "{% if backup_db_type | default('mariadb') == 'postgres' %}pg_dump{% elif backup_db_type | default('mariadb') == 'mysql' %}mysqldump{% else %}mariadb-dump{% endif %} {{ db_container }} exited $dump_rc"
|
||||||
|
|
||||||
# A new dump is promoted over yesterday's only after it proves complete:
|
# A new dump is promoted over yesterday's only after it proves complete: a
|
||||||
# a valid gzip stream AND the "-- Dump completed" trailer that mariadb-dump
|
# valid gzip stream AND the trailer the dump tool writes only on a clean
|
||||||
# writes only on a clean finish. `mv` is atomic within the staging
|
# finish. The two engines word it differently, so the string is per-type --
|
||||||
# filesystem, so a failed or truncated run can never replace a good dump.
|
# grepping for the MariaDB one against a pg_dump would fail every run.
|
||||||
|
# `mv` is atomic within the staging filesystem, so a failed or truncated run
|
||||||
|
# can never replace a good dump.
|
||||||
gzip -t "$tmp" || fail "dump is not a valid gzip stream"
|
gzip -t "$tmp" || fail "dump is not a valid gzip stream"
|
||||||
gunzip -c "$tmp" | tail -c 512 | grep -q 'Dump completed' \
|
gunzip -c "$tmp" | tail -c 512 \
|
||||||
|
| grep -q '{{ "PostgreSQL database dump complete" if backup_db_type | default("mariadb") == "postgres" else "Dump completed" }}' \
|
||||||
|| fail "dump is truncated (no completion trailer)"
|
|| fail "dump is truncated (no completion trailer)"
|
||||||
mv -f "$tmp" "$DUMP"
|
mv -f "$tmp" "$DUMP"
|
||||||
log "db_dump=ok bytes=$(stat -c %s "$DUMP")"
|
log "db_dump=ok bytes=$(stat -c %s "$DUMP")"
|
||||||
@@ -87,6 +119,32 @@ log "db_dump=ok bytes=$(stat -c %s "$DUMP")"
|
|||||||
# periodic ZFS snapshots (see roles/podman/README.md).
|
# periodic ZFS snapshots (see roles/podman/README.md).
|
||||||
ls -1t "$STAGE"/db/"$INSTANCE"-*.sql.gz | tail -n +$((KEEP + 1)) | xargs -r rm -f
|
ls -1t "$STAGE"/db/"$INSTANCE"-*.sql.gz | tail -n +$((KEEP + 1)) | xargs -r rm -f
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
{% if backup_sqlite_dbs | default([]) %}
|
||||||
|
# ----------------------------------------------------- 1b. sqlite snapshots
|
||||||
|
# These databases are WAL-mode SQLite written by a LIVE process, so neither
|
||||||
|
# rsync option is correct: skipping the -wal loses committed transactions,
|
||||||
|
# and copying .db + -wal together can catch a checkpoint mid-write. Only
|
||||||
|
# `.backup` takes a consistent snapshot of a database that is being written.
|
||||||
|
#
|
||||||
|
# Run on the HOST, not in the container: the files live on the host
|
||||||
|
# filesystem and sqlite3 is installed there, so this needs no podman at all
|
||||||
|
# and works even when the owning container is stopped.
|
||||||
|
install -d -m 0700 "$STAGE" "$STAGE/db"
|
||||||
|
{% for db in backup_sqlite_dbs %}
|
||||||
|
sqlite_out="$STAGE/db/{{ db | basename | regex_replace('\\.db$', '') }}.sqlite"
|
||||||
|
if [ ! -f "{{ db }}" ]; then
|
||||||
|
fail "sqlite source missing: {{ db }}"
|
||||||
|
fi
|
||||||
|
sqlite3 "{{ db }}" ".backup '$sqlite_out.tmp'" \
|
||||||
|
|| fail "sqlite3 .backup failed for {{ db }}"
|
||||||
|
# Prove the snapshot is a loadable database before it replaces yesterday's,
|
||||||
|
# mirroring the gzip/trailer gate the SQL dumps get above.
|
||||||
|
sqlite3 "$sqlite_out.tmp" 'pragma integrity_check;' | grep -qx ok \
|
||||||
|
|| fail "sqlite snapshot failed integrity_check: {{ db }}"
|
||||||
|
mv -f "$sqlite_out.tmp" "$sqlite_out"
|
||||||
|
log "sqlite_dump=ok db={{ db | basename }} bytes=$(stat -c %s "$sqlite_out")"
|
||||||
|
{% endfor %}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
# ----------------------------------------------------------- 2. file tree
|
# ----------------------------------------------------------- 2. file tree
|
||||||
# --exclude .ssh is load-bearing: {{ remote_path }} IS {{ ssh_user }}'s home
|
# --exclude .ssh is load-bearing: {{ remote_path }} IS {{ ssh_user }}'s home
|
||||||
@@ -108,7 +166,7 @@ log "syncing config"
|
|||||||
rsync -az --timeout=600 --delete --mkpath {{ backup_rsync_extra_args | default('') }} \
|
rsync -az --timeout=600 --delete --mkpath {{ backup_rsync_extra_args | default('') }} \
|
||||||
-e "$SSH" {{ config_path }}/ "$DEST:{{ remote_path }}/_backup/config/"
|
-e "$SSH" {{ config_path }}/ "$DEST:{{ remote_path }}/_backup/config/"
|
||||||
{% endif %}
|
{% endif %}
|
||||||
{% if db_container | default('') %}
|
{% if db_container | default('') or backup_sqlite_dbs | default([]) %}
|
||||||
log "syncing db dumps"
|
log "syncing db dumps"
|
||||||
rsync -az --timeout=600 --delete --mkpath {{ backup_rsync_extra_args | default('') }} \
|
rsync -az --timeout=600 --delete --mkpath {{ backup_rsync_extra_args | default('') }} \
|
||||||
-e "$SSH" "$STAGE/db/" "$DEST:{{ remote_path }}/_backup/db/"
|
-e "$SSH" "$STAGE/db/" "$DEST:{{ remote_path }}/_backup/db/"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Daily Nextcloud {{ instance_name }} backup
|
Description=Daily {{ backup_product | default('Nextcloud') }} {{ instance_name }} backup
|
||||||
|
|
||||||
[Timer]
|
[Timer]
|
||||||
OnCalendar={{ backup_oncalendar | default('*-*-* 04:00:00') }}
|
OnCalendar={{ backup_oncalendar | default('*-*-* 04:00:00') }}
|
||||||
|
|||||||
@@ -1,8 +1,12 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# {{ ansible_managed }}
|
# {{ ansible_managed }}
|
||||||
# OnFailure= handler for the Nextcloud backup units. Invoked as:
|
# OnFailure= handler for every backup unit (Nextcloud and Gitea). Invoked as:
|
||||||
# nextcloud-backup-alert.sh <failed-unit-name>
|
# nextcloud-backup-alert.sh <failed-unit-name>
|
||||||
#
|
#
|
||||||
|
# One shared copy serves all instances -- cloud-backup.yml renders it once and
|
||||||
|
# later includes are no-ops -- so the wording here must not name one product.
|
||||||
|
# The failing UNIT name is what identifies the instance.
|
||||||
|
#
|
||||||
# Deliberately NOT `set -e`: an alert handler that dies partway through
|
# Deliberately NOT `set -e`: an alert handler that dies partway through
|
||||||
# reports nothing, which is worse than a partial report. Same reasoning as
|
# reports nothing, which is worse than a partial report. Same reasoning as
|
||||||
# roles/ups/templates/ups-restore.sh.j2.
|
# roles/ups/templates/ups-restore.sh.j2.
|
||||||
@@ -23,13 +27,13 @@ code="$(systemctl show -p ExecMainStatus --value "$UNIT" 2>/dev/null)"
|
|||||||
if [ "${result:-success}" = "success" ]; then
|
if [ "${result:-success}" = "success" ]; then
|
||||||
state=spurious
|
state=spurious
|
||||||
prio=daemon.warning
|
prio=daemon.warning
|
||||||
headline="$(printf 'Nextcloud backup alert handler was invoked on %s, but %s reports SUCCESS.\nThis is not a backup failure -- most likely the handler was started manually.' "$HOST" "$UNIT")"
|
headline="$(printf 'Backup alert handler was invoked on %s, but %s reports SUCCESS.\nThis is not a backup failure -- most likely the handler was started manually.' "$HOST" "$UNIT")"
|
||||||
subject="[$HOST] Nextcloud backup alert (spurious, unit OK): $UNIT"
|
subject="[$HOST] Backup alert (spurious, unit OK): $UNIT"
|
||||||
else
|
else
|
||||||
state=failed
|
state=failed
|
||||||
prio=daemon.err
|
prio=daemon.err
|
||||||
headline="Nextcloud backup FAILED on $HOST"
|
headline="Backup FAILED on $HOST"
|
||||||
subject="[$HOST] Nextcloud backup FAILED: $UNIT"
|
subject="[$HOST] Backup FAILED: $UNIT"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# One machine-parseable line for Graylog, then the context.
|
# One machine-parseable line for Graylog, then the context.
|
||||||
|
|||||||
Binary file not shown.
Reference in New Issue
Block a user