bound log growth and reclaim ~52 GB of container disk

Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not
holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across
16 log streams. Made explicit at 10MiB/keep 3/7d.

Note roll_size et al are subdirectives of `output file`, NOT of `log`.
Getting that wrong does not degrade gracefully: Caddy refuses to start on a
bad config, so every site went down until it was corrected. Worth a
`caddy validate` gate before reload.

journald had no SystemMaxUse and had reached 4 GB, drifting toward its
10%-of-filesystem default (~190 GB on this root). Capped at 500M.

Both are safe to keep short because fluent-bit ships the journal and every
Caddy access log into Graylog -- though note its GELF output has been
erroring for days, which weakens that premise and wants investigating.

The larger find was unrelated to logs: 896 images totalling 59.6 GB with
75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and
5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees
orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added
a weekly timer keeping 30 days so a rollback still needs no rebuild.

Also dropped the decommissioned 6379/tcp redis rule (nothing listening;
Immich's redis is on the shared podman network) and the orphaned nosql, s3
and searxng volume dirs.

Backup log exclusions turned out to be unnecessary: Gitea logs to console
so its log dirs are empty, Nextcloud already excludes its own, BookStack
mounts only uploads, and Caddy is not backed up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
2026-07-31 11:13:35 -04:00
parent bc110ce69e
commit f11391b28f
11 changed files with 260 additions and 18 deletions
+19
View File
@@ -134,6 +134,16 @@ caddy_local_networks:
caddy_log_level: INFO
caddy_log_format: json
# Log rotation. Caddy rotates on implicit defaults (100MiB / keep 10 / 90d)
# without these, and those were demonstrably not holding: 20 rotated files per
# stream for the busiest logs and a 190-day-old .gz, for 1.3 GB across 16
# streams. Kept short deliberately -- fluent-bit tails every one of these into
# Graylog (see roles/common/templates/fluent-bit/fluent-bit.conf.j2), so
# Graylog is the system of record and the local files are only a buffer.
caddy_log_roll_size: 10MiB
caddy_log_roll_keep: 3
caddy_log_roll_keep_for: 168h
# Caddy performance tuning
caddy_max_request_body_mb: 500
@@ -175,3 +185,12 @@ geoip_path: "{{ graylog_path }}/geoip"
geoip_database_edition: GeoLite2-City
# geoip_maxmind_account_id: defined in vault
# geoip_maxmind_license_key: defined in vault
# Weekly podman prune (see templates/podman-prune.sh.j2). Both rootless users
# have their own image store; the git user runs the Gitea pods.
podman_prune_users:
- "{{ podman_user }}"
- "{{ git_user }}"
# Keep 30 days of unused images so a rollback needs no rebuild or re-pull.
podman_prune_until: 720h
podman_prune_oncalendar: "Sun *-*-* 02:00:00"