# ups

UPS monitoring and staged shutdown for the home rack.

A CyberPower PR1500RT2U (`0764:0601`) is cabled by USB to `home.debyl.io` and
backs both that host and `truenas.localdomain` (Dell PowerEdge R415). This role
makes `home.debyl.io` the NUT server and gives it the ability to power TrueNAS
back on over IPMI.

## Outage sequence

| When | What happens | Driven by |
| --- | --- | --- |
| t+0 | UPS goes on battery, `ONBATT` logged to journald → Graylog | `upsmon` |
| t+2min | TrueNAS shuts itself down cleanly, shedding ~200 W | TrueNAS UPS service, Slave mode, `Shutdown Timer 120` |
| 10% charge | `home.debyl.io` shuts itself down and tells the UPS to cut its output | `upsmon` `SHUTDOWNCMD` + `/lib/systemd/system-shutdown/nutshutdown` |
| mains returns | UPS re-energizes; both machines power themselves back up | R415 `always-on` restore policy; `home.debyl.io` BIOS *After Power Loss → Power On* |
| mains back +5min | Best-effort IPMI power-on, if a dedicated iDRAC is ever fitted | `upssched` → `ups-restore.sh` |
| host boot | Same restore check, for the deep-drain case | `ups-restore.service` |

The 10% threshold is not a custom poller. CyberPower asserts its own low-battery
flag around 20–35%, so `ups.conf` sets `ignorelb` plus
`override.battery.charge.low`, and stock `upsmon` fires at exactly the
configured percentage.

Likewise, the 2-minute TrueNAS shed is TrueNAS's own native "shutdown timer"
setting — no SSH key and no shutdown script from this side.

## Powering TrueNAS back on

Neither Wake-on-LAN nor IPMI works on this box today, so restore is done with
the chassis power restore policy instead.

**Wake-on-LAN is out.** `bce0` advertises no WOL capability (`ifconfig -m bce0`
has no `WOL_MAGIC`) and both NICs are bonded into an LACP `lagg0`.

**IPMI is out too, for now.** The iDRAC is in `shared / LOM1` mode and the
Enterprise card that would provide a dedicated management port is not fitted:

```
$ ipmitool sdr elist | grep -i idrac
iDRAC6 Ent Pres  | 70h | ok  |  7.1 | Absent
```

A shared-LOM iDRAC6 Express has no standby power. Measured directly: with the
chassis powered off the BMC does not even answer ARP, and it only reappears
~200s into POST, at the moment the host brings the NIC link up. That is a
hardware limitation, not a switch or BIOS problem.

**So restore works like this instead.** `ipmitool chassis policy always-on` is
set on the R415. On a deep outage `home.debyl.io` halts at 10% and NUT's
shutdown hook tells the UPS to cut its output; when mains returns the UPS
re-energizes, the R415 sees AC and boots itself.

The gap is the medium outage — mains returns after TrueNAS has shed but before
the battery reaches 10%. The UPS never cuts power, so TrueNAS stays off and
needs a manual power button press. Fitting a used iDRAC6 Enterprise card and
running `ipmitool delloem lan set dedicated` (plus a cable to the dedicated
port) closes that gap, and `ups-restore.sh` starts working with no code
changes — it is already deployed and simply logs and exits while the BMC is
unreachable.

## One-time setup outside Ansible

These are not managed by this role.

**iDRAC (already done, via `ipmitool` on TrueNAS):**

```
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 192.168.1.12
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.1
ipmitool lan set 1 access on               # was disabled - nothing answers without this
ipmitool channel setaccess 1 2 callin=on ipmi=on link=on privilege=4
ipmitool user set password 2 '<idrac_password>'
ipmitool chassis policy always-on          # this is what restores power after an outage
```

`idrac_password` is also the iDRAC web UI password for `root` - they share a
user database.

**TrueNAS UI → Services → UPS** (enable + start automatically):

| Field | Value |
| --- | --- |
| UPS Mode | Slave |
| Remote Host | `192.168.1.10` |
| Remote Port | `3493` |
| Identifier | `cyberpower` |
| Monitor User | `truenas` |
| Monitor Password | vault `nut_truenas_password` |
| Shutdown Mode | UPS goes on battery |
| Shutdown Timer | `120` |
| Shutdown Command | `/sbin/shutdown -p now` |
| Power Off UPS | unchecked |

**BIOS on `home.debyl.io`** (Lenovo 10MR0004US): Power → *After Power Loss* →
**Power On**. On a full drain the NUT shutdown hook
(`/lib/systemd/system-shutdown/nutshutdown`) tells the UPS to cut its own
output; this BIOS setting is what brings the host back when mains returns.

## Vault keys

`nut_upsmon_password`, `nut_truenas_password`, `idrac_password`

## Operating

```
upsc cyberpower                      # full UPS status
upsc cyberpower battery.charge
sudo /usr/local/bin/truenas-power.sh status|on|soft|off
journalctl -t ups-restore -t ups-sched
```