Bastian de BylandClaude Opus 5
9954d774e7
After the host rebooted, truenas began mailing NOCOMM continuously. The UPS driver was fine throughout -- nut-driver@cyberpower stayed running and the CyberPower is still on USB here. What died was upsd, which publishes its state to truenas over the network: upsd: not listening on 192.168.1.10 port 3493 upsd: Fatal error: some listening interfaces were not available nut-server.service: Start request repeated too quickly upsd binds an explicit address, but the packaged unit only orders itself After=network.target, which is satisfied when networking STARTS rather than when an address exists. It tried to bind 11 seconds into boot, before NetworkManager had assigned the address, then burned all five default restart attempts inside one second, tripped the start limit and stayed dead. The shipped unit carries these two lines commented out, because upstream knows the case. network-online.target is the correct ordering; NetworkManager-wait-online is enabled here so it genuinely waits for addresses. StartLimitIntervalSec=0 and RestartSec are belt and braces: a slow address can no longer exhaust the attempts, and retries are spaced instead of hammered. truenas needs no change -- it is a correctly configured SLAVE that reconnected on its own, and its NUT config is regenerated from the middleware database anyway. Verified: upsd listening on both addresses, UPS OL at 100%, and truenas querying it again with zero failures since. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ups
UPS monitoring and staged shutdown for the home rack.
A CyberPower PR1500RT2U (0764:0601) is cabled by USB to home.debyl.io and
backs both that host and truenas.localdomain (Dell PowerEdge R415). This role
makes home.debyl.io the NUT server and gives it the ability to power TrueNAS
back on over IPMI.
Outage sequence
| When | What happens | Driven by |
|---|---|---|
| t+0 | UPS goes on battery, ONBATT logged to journald → Graylog |
upsmon |
| t+2min | TrueNAS shuts itself down cleanly, shedding ~200 W | TrueNAS UPS service, Slave mode, Shutdown Timer 120 |
| 10% charge | home.debyl.io shuts itself down and tells the UPS to cut its output |
upsmon SHUTDOWNCMD + /lib/systemd/system-shutdown/nutshutdown |
| mains returns | UPS re-energizes; both machines power themselves back up | R415 always-on restore policy; home.debyl.io BIOS After Power Loss → Power On |
| mains back +5min | Best-effort IPMI power-on, if a dedicated iDRAC is ever fitted | upssched → ups-restore.sh |
| host boot | Same restore check, for the deep-drain case | ups-restore.service |
The 10% threshold is not a custom poller. CyberPower asserts its own low-battery
flag around 20–35%, so ups.conf sets ignorelb plus
override.battery.charge.low, and stock upsmon fires at exactly the
configured percentage.
Likewise, the 2-minute TrueNAS shed is TrueNAS's own native "shutdown timer" setting — no SSH key and no shutdown script from this side.
Powering TrueNAS back on
Neither Wake-on-LAN nor IPMI works on this box today, so restore is done with the chassis power restore policy instead.
Wake-on-LAN is out. bce0 advertises no WOL capability (ifconfig -m bce0
has no WOL_MAGIC) and both NICs are bonded into an LACP lagg0.
IPMI is out too, for now. The iDRAC is in shared / LOM1 mode and the
Enterprise card that would provide a dedicated management port is not fitted:
$ ipmitool sdr elist | grep -i idrac
iDRAC6 Ent Pres | 70h | ok | 7.1 | Absent
A shared-LOM iDRAC6 Express has no standby power. Measured directly: with the chassis powered off the BMC does not even answer ARP, and it only reappears ~200s into POST, at the moment the host brings the NIC link up. That is a hardware limitation, not a switch or BIOS problem.
So restore works like this instead. ipmitool chassis policy always-on is
set on the R415. On a deep outage home.debyl.io halts at 10% and NUT's
shutdown hook tells the UPS to cut its output; when mains returns the UPS
re-energizes, the R415 sees AC and boots itself.
The gap is the medium outage — mains returns after TrueNAS has shed but before
the battery reaches 10%. The UPS never cuts power, so TrueNAS stays off and
needs a manual power button press. Fitting a used iDRAC6 Enterprise card and
running ipmitool delloem lan set dedicated (plus a cable to the dedicated
port) closes that gap, and ups-restore.sh starts working with no code
changes — it is already deployed and simply logs and exits while the BMC is
unreachable.
One-time setup outside Ansible
These are not managed by this role.
iDRAC (already done, via ipmitool on TrueNAS):
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 192.168.1.12
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.1
ipmitool lan set 1 access on # was disabled - nothing answers without this
ipmitool channel setaccess 1 2 callin=on ipmi=on link=on privilege=4
ipmitool user set password 2 '<idrac_password>'
ipmitool chassis policy always-on # this is what restores power after an outage
idrac_password is also the iDRAC web UI password for root - they share a
user database.
TrueNAS UI → Services → UPS (enable + start automatically):
| Field | Value |
|---|---|
| UPS Mode | Slave |
| Remote Host | 192.168.1.10 |
| Remote Port | 3493 |
| Identifier | cyberpower |
| Monitor User | truenas |
| Monitor Password | vault nut_truenas_password |
| Shutdown Mode | UPS goes on battery |
| Shutdown Timer | 120 |
| Shutdown Command | /sbin/shutdown -p now |
| Power Off UPS | unchecked |
BIOS on home.debyl.io (Lenovo 10MR0004US): Power → After Power Loss →
Power On. On a full drain the NUT shutdown hook
(/lib/systemd/system-shutdown/nutshutdown) tells the UPS to cut its own
output; this BIOS setting is what brings the host back when mains returns.
Vault keys
nut_upsmon_password, nut_truenas_password, idrac_password
Operating
upsc cyberpower # full UPS status
upsc cyberpower battery.charge
sudo /usr/local/bin/truenas-power.sh status|on|soft|off
journalctl -t ups-restore -t ups-sched