bound log growth and reclaim ~52 GB of container disk
Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across 16 log streams. Made explicit at 10MiB/keep 3/7d. Note roll_size et al are subdirectives of `output file`, NOT of `log`. Getting that wrong does not degrade gracefully: Caddy refuses to start on a bad config, so every site went down until it was corrected. Worth a `caddy validate` gate before reload. journald had no SystemMaxUse and had reached 4 GB, drifting toward its 10%-of-filesystem default (~190 GB on this root). Capped at 500M. Both are safe to keep short because fluent-bit ships the journal and every Caddy access log into Graylog -- though note its GELF output has been erroring for days, which weakens that premise and wants investigating. The larger find was unrelated to logs: 896 images totalling 59.6 GB with 75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and 5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added a weekly timer keeping 30 days so a rollback still needs no rebuild. Also dropped the decommissioned 6379/tcp redis rule (nothing listening; Immich's redis is on the shared podman network) and the orphaned nosql, s3 and searxng volume dirs. Backup log exclusions turned out to be unnecessary: Gitea logs to console so its log dirs are empty, Nextcloud already excludes its own, BookStack mounts only uploads, and Caddy is not backed up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -134,6 +134,16 @@ caddy_local_networks:
|
||||
caddy_log_level: INFO
|
||||
caddy_log_format: json
|
||||
|
||||
# Log rotation. Caddy rotates on implicit defaults (100MiB / keep 10 / 90d)
|
||||
# without these, and those were demonstrably not holding: 20 rotated files per
|
||||
# stream for the busiest logs and a 190-day-old .gz, for 1.3 GB across 16
|
||||
# streams. Kept short deliberately -- fluent-bit tails every one of these into
|
||||
# Graylog (see roles/common/templates/fluent-bit/fluent-bit.conf.j2), so
|
||||
# Graylog is the system of record and the local files are only a buffer.
|
||||
caddy_log_roll_size: 10MiB
|
||||
caddy_log_roll_keep: 3
|
||||
caddy_log_roll_keep_for: 168h
|
||||
|
||||
# Caddy performance tuning
|
||||
caddy_max_request_body_mb: 500
|
||||
|
||||
@@ -175,3 +185,12 @@ geoip_path: "{{ graylog_path }}/geoip"
|
||||
geoip_database_edition: GeoLite2-City
|
||||
# geoip_maxmind_account_id: defined in vault
|
||||
# geoip_maxmind_license_key: defined in vault
|
||||
|
||||
# Weekly podman prune (see templates/podman-prune.sh.j2). Both rootless users
|
||||
# have their own image store; the git user runs the Gitea pods.
|
||||
podman_prune_users:
|
||||
- "{{ podman_user }}"
|
||||
- "{{ git_user }}"
|
||||
# Keep 30 days of unused images so a rollback needs no rebuild or re-pull.
|
||||
podman_prune_until: 720h
|
||||
podman_prune_oncalendar: "Sun *-*-* 02:00:00"
|
||||
|
||||
Reference in New Issue
Block a user