Bastian de Byl
f11391b28f
Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across 16 log streams. Made explicit at 10MiB/keep 3/7d. Note roll_size et al are subdirectives of `output file`, NOT of `log`. Getting that wrong does not degrade gracefully: Caddy refuses to start on a bad config, so every site went down until it was corrected. Worth a `caddy validate` gate before reload. journald had no SystemMaxUse and had reached 4 GB, drifting toward its 10%-of-filesystem default (~190 GB on this root). Capped at 500M. Both are safe to keep short because fluent-bit ships the journal and every Caddy access log into Graylog -- though note its GELF output has been erroring for days, which weakens that premise and wants investigating. The larger find was unrelated to logs: 896 images totalling 59.6 GB with 75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and 5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added a weekly timer keeping 30 days so a rollback still needs no rebuild. Also dropped the decommissioned 6379/tcp redis rule (nothing listening; Immich's redis is on the shared podman network) and the orphaned nosql, s3 and searxng volume dirs. Backup log exclusions turned out to be unnecessary: Gitea logs to console so its log dirs are empty, Nextcloud already excludes its own, BookStack mounts only uploads, and Caddy is not backed up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13 lines
649 B
Django/Jinja
13 lines
649 B
Django/Jinja
### {{ ansible_managed }}
|
|
# Without a cap journald sizes itself at 10% of the filesystem, which on this
|
|
# 1.9 TB root means it will happily grow into the tens of gigabytes; it had
|
|
# reached 4 GB before this was set.
|
|
#
|
|
# Kept small on purpose: every container runs with log_driver=journald and
|
|
# fluent-bit drains the journal into Graylog continuously (systemd input,
|
|
# _COMM=conmon -- see templates/fluent-bit/fluent-bit.conf.j2), so Graylog is
|
|
# the system of record. What stays here is only the buffer that covers
|
|
# fluent-bit being down, and 500M is a long outage at this log rate.
|
|
[Journal]
|
|
SystemMaxUse={{ journald_max_use | default('500M') }}
|