UPS monitoring, Nextcloud cron, container image bumps

ups role: NUT server on home.debyl.io for the CyberPower PR1500RT2U that backs both it and truenas.localdomain, with staged shutdown (TrueNAS sheds at t+2min, host at 10% charge) and best-effort IPMI power-on when mains returns. The 10% threshold leans on ignorelb + override.battery.charge.low rather than a custom poller, because CyberPower asserts its own low-battery flag far too early. Credentials come from vault vars; nothing sensitive is templated in the clear. Nextcloud background jobs: both instances have backgroundjobs_mode "cron", which expects an external caller every ~5 minutes, and nothing was calling. The personal instance had not run a background job since 2026-05-14 and skudak since 2024-11-20. Consequently trash and file versions never expired, stale chunked uploads accumulated, calendar reminders never fired, and nextcloud.log was never rotated -- which quietly made the existing log_rotate_size cap inert. Added a systemd timer per instance, skipping cleanly when the container is down or in maintenance so deploy windows don't show up as failed units. Trash retention on the personal instance: the default "auto" only expires when disk space demands it, so 66 GB of >30-day deletions sat on a host with 1.3 TB free -- effectively unbounded. "auto, 30" makes the 30-day expiry unconditional while still purging early under pressure. Image bumps: nextcloud 33.0.0 -> 34.0.2 (both cloud and skudak-cloud) greg-time-bot 3.9.25 -> 3.10.0 fulfillr 20260723.2044 -> 20260728.2155 (prod and dev) The fulfillr bump records what is already deployed: both containers were rolled to that image on 2026-07-29 for SCRUM-156 (digital product releases + customer update campaign). Committing it keeps the repo from claiming an older tag than the host is actually running, which would otherwise roll fulfillr backwards on the next clean-checkout deploy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 13:00:51 -04:00
parent 16145fb6bd
commit 5776dbe1bf
27 changed files with 768 additions and 5 deletions
@@ -0,0 +1,3 @@
+# {{ ansible_managed }}
+# netserver: run the driver + upsd locally and serve slaves over the network.
+MODE=netserver
@@ -0,0 +1,34 @@
+#!/bin/bash
+# {{ ansible_managed }}
+# Remote power control for truenas.localdomain via the R415 iDRAC6.
+# Usage: truenas-power.sh {status|on|soft|off|cycle}
+set -euo pipefail
+
+PW_FILE=/etc/ups/idrac.pw
+
+if [ ! -r "$PW_FILE" ]; then
+    echo "cannot read $PW_FILE" >&2
+    exit 1
+fi
+
+# -f keeps the password out of the process argument list.
+ipmi() {
+    ipmitool -I lanplus -H {{ idrac_host }} -U {{ idrac_user }} \
+        -f "$PW_FILE" "$@"
+}
+
+case "${1:-status}" in
+    status) ipmi chassis power status ;;
+    on)     ipmi chassis power on ;;
+    # ACPI soft-off. FreeBSD has hw.acpi.power_button_state=S5, so this is
+    # a clean TrueNAS shutdown. Normally unused: TrueNAS shuts itself down
+    # as a NUT slave. This is the manual escape hatch.
+    soft)   ipmi chassis power soft ;;
+    # Hard cut, last resort only.
+    off)    ipmi chassis power off ;;
+    cycle)  ipmi chassis power cycle ;;
+    *)
+        echo "usage: $0 {status|on|soft|off|cycle}" >&2
+        exit 2
+        ;;
+esac
@@ -0,0 +1,13 @@
+# {{ ansible_managed }}
+[Unit]
+Description=Restore TrueNAS power after an outage
+After=network-online.target nut-server.service nut-monitor.service
+Wants=network-online.target
+Requires=nut-server.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/ups-restore.sh
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,63 @@
+#!/bin/bash
+# {{ ansible_managed }}
+# Power truenas.localdomain back on after an outage, but only when it is
+# genuinely safe to do so. Called by upssched once mains has been stable,
+# and once at boot by ups-restore.service (deep-drain recovery).
+#
+# Deliberately stateless: we ask the iDRAC whether the chassis is off
+# rather than tracking whether we were the ones who shut it down. A NAS
+# powered off by hand while on mains is safe, because no ONLINE event
+# fires in that case and the boot path checks UPS status first.
+#
+# NOTE: as of now this is a best-effort secondary path. The R415 has no
+# iDRAC6 Enterprise card ("iDRAC6 Ent Pres ... Absent"), so its shared-LOM
+# BMC has no standby power and goes unreachable whenever the chassis is
+# off - exactly when we would want it. An unreachable iDRAC is therefore
+# the EXPECTED case here, and we exit quietly rather than alarming.
+#
+# The primary restore path needs no IPMI: the R415 power restore policy is
+# set to always-on, so when the UPS cuts and then restores its output the
+# server powers itself back up. Fit an iDRAC6 Enterprise card and switch it
+# to dedicated mode and this script starts working with no changes.
+set -uo pipefail
+
+TAG=ups-restore
+UPS={{ ups_name }}@localhost
+
+log() { logger -t "$TAG" -- "$*"; echo "$TAG: $*"; }
+
+# At boot, upsd may not be serving yet. Wait a bounded amount of time.
+for _ in $(seq 1 30); do
+    if upsc "$UPS" ups.status >/dev/null 2>&1; then
+        break
+    fi
+    sleep 2
+done
+
+status=$(upsc "$UPS" ups.status 2>/dev/null || echo UNKNOWN)
+if [[ "$status" != *OL* ]]; then
+    log "UPS status is '$status', not on line - refusing to power TrueNAS on"
+    exit 0
+fi
+
+power=$(/usr/local/bin/truenas-power.sh status 2>&1 || true)
+case "$power" in
+    *"is off"*)
+        log "mains stable and chassis off - powering TrueNAS on"
+        if /usr/local/bin/truenas-power.sh on; then
+            log "power-on command accepted"
+        else
+            log "power-on command FAILED - check iDRAC at {{ idrac_host }}"
+            exit 1
+        fi
+        ;;
+    *"is on"*)
+        log "TrueNAS already on, nothing to do"
+        ;;
+    *)
+        # Expected while the box has no iDRAC6 Enterprise card: the BMC is
+        # simply not on the network with the chassis powered down.
+        log "iDRAC at {{ idrac_host }} unreachable - relying on the" \
+            "always-on power restore policy instead ($power)"
+        ;;
+esac
@@ -0,0 +1,14 @@
+#!/bin/bash
+# {{ ansible_managed }}
+# upssched CMDSCRIPT. Runs as the unprivileged nut user.
+set -uo pipefail
+
+case "${1:-}" in
+    truenas-restore)
+        exec /usr/local/bin/ups-restore.sh
+        ;;
+    *)
+        logger -t ups-sched -- "unknown timer '${1:-}'"
+        exit 1
+        ;;
+esac
@@ -0,0 +1,18 @@
+# {{ ansible_managed }}
+
+[{{ ups_name }}]
+    driver = usbhid-ups
+    port = auto
+    vendorid = {{ ups_vendorid }}
+    productid = {{ ups_productid }}
+    desc = "{{ ups_desc }}"
+    # CyberPower asserts its own low-battery flag around 20-35%, far too
+    # early for us. Ignore it and derive LB from our own threshold so
+    # upsmon fires SHUTDOWNCMD at exactly {{ ups_low_charge_pct }}%.
+    ignorelb
+    override.battery.charge.low = {{ ups_low_charge_pct }}
+    # Unlock driver.killpower. Without this the driver refuses to cut UPS
+    # output at shutdown, and the R415's always-on power restore policy
+    # would never see AC drop and return - i.e. nothing comes back after a
+    # deep outage. See README.md.
+    allow_killpower
@@ -0,0 +1,3 @@
+# {{ ansible_managed }}
+LISTEN 127.0.0.1 {{ ups_listen_port }}
+LISTEN {{ ups_listen_addr }} {{ ups_listen_port }}
@@ -0,0 +1,11 @@
+# {{ ansible_managed }}
+
+# Local upsmon on this host.
+[upsmon]
+    password = {{ nut_upsmon_password }}
+    upsmon master
+
+# truenas.localdomain, running the TrueNAS UPS service in Slave mode.
+[truenas]
+    password = {{ nut_truenas_password }}
+    upsmon slave
@@ -0,0 +1,40 @@
+# {{ ansible_managed }}
+
+MONITOR {{ ups_name }}@localhost 1 upsmon {{ nut_upsmon_password }} master
+
+MINSUPPLIES 1
+SHUTDOWNCMD "/usr/bin/systemctl poweroff"
+NOTIFYCMD /usr/bin/upssched
+
+# upsmon drops this file before halting; /lib/systemd/system-shutdown/nutshutdown
+# reads it late in shutdown and, if present, tells the UPS to cut its output.
+# That AC drop-and-return is what triggers the R415's always-on restore policy
+# and this host's BIOS "After Power Loss: Power On". upsmon has NO compiled-in
+# default for this - leave it unset and the UPS never powers down.
+# Must be on tmpfs: a persistent path can go stale and make every ordinary
+# reboot look like a forced shutdown.
+POWERDOWNFLAG /run/nut/killpower
+
+POLLFREQ 5
+POLLFREQALERT 5
+
+# Wait up to 30s for the truenas slave to disconnect before we halt.
+HOSTSYNC 30
+DEADTIME 15
+RBWARNTIME 43200
+NOCOMMWARNTIME 300
+FINALDELAY 5
+
+# SYSLOG puts every UPS event in the journal, which fluent-bit already
+# forwards to Graylog (see roles/common/tasks/fluent-bit.yml).
+# EXEC runs NOTIFYCMD, i.e. upssched, which drives the TrueNAS restore.
+NOTIFYFLAG ONLINE    SYSLOG+EXEC
+NOTIFYFLAG ONBATT    SYSLOG+EXEC
+NOTIFYFLAG LOWBATT   SYSLOG+EXEC
+NOTIFYFLAG FSD       SYSLOG+EXEC
+NOTIFYFLAG COMMOK    SYSLOG+EXEC
+NOTIFYFLAG COMMBAD   SYSLOG+EXEC
+NOTIFYFLAG SHUTDOWN  SYSLOG+EXEC
+NOTIFYFLAG REPLBATT  SYSLOG
+NOTIFYFLAG NOCOMM    SYSLOG
+NOTIFYFLAG NOPARENT  SYSLOG
@@ -0,0 +1,13 @@
+# {{ ansible_managed }}
+
+CMDSCRIPT /usr/local/bin/ups-sched-cmd.sh
+PIPEFN /run/nut/upssched.pipe
+LOCKFN /run/nut/upssched.lock
+
+# Mains is back: wait for it to hold for {{ ups_restore_stable_secs }}s
+# before powering TrueNAS back on, so we do not flap on unstable power.
+AT ONLINE * CANCEL-TIMER truenas-restore
+AT ONLINE * START-TIMER truenas-restore {{ ups_restore_stable_secs }}
+
+# Power dropped again while the restore timer was pending - stand down.
+AT ONBATT * CANCEL-TIMER truenas-restore