bound log growth and reclaim ~52 GB of container disk

Caddy was rotating on implicit defaults (100MiB/keep 10/90d) that were not
holding -- 20 rotated files per stream and a 190-day-old .gz, 1.3 GB across
16 log streams. Made explicit at 10MiB/keep 3/7d.

Note roll_size et al are subdirectives of `output file`, NOT of `log`.
Getting that wrong does not degrade gracefully: Caddy refuses to start on a
bad config, so every site went down until it was corrected. Worth a
`caddy validate` gate before reload.

journald had no SystemMaxUse and had reached 4 GB, drifting toward its
10%-of-filesystem default (~190 GB on this root). Capped at 500M.

Both are safe to keep short because fluent-bit ships the journal and every
Caddy access log into Graylog -- though note its GELF output has been
erroring for days, which weakens that premise and wants investigating.

The larger find was unrelated to logs: 896 images totalling 59.6 GB with
75% unused (94 tags of greg-time-bot, 73 of fulfillr -- one per deploy) and
5.4 GB of dangling volumes, mostly 804 MB Nextcloud /var/www/html trees
orphaned by container recreations. Pruned to 22 images / 15.4 GB, and added
a weekly timer keeping 30 days so a rollback still needs no rebuild.

Also dropped the decommissioned 6379/tcp redis rule (nothing listening;
Immich's redis is on the shared podman network) and the orphaned nosql, s3
and searxng volume dirs.

Backup log exclusions turned out to be unnecessary: Gitea logs to console
so its log dirs are empty, Nextcloud already excludes its own, BookStack
mounts only uploads, and Caddy is not backed up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
2026-07-31 11:13:35 -04:00
parent bc110ce69e
commit f11391b28f
11 changed files with 260 additions and 18 deletions
-2
View File
@@ -15,8 +15,6 @@
- 443/tcp
# Gitea Skudak SSH
- 2222/tcp
# nosql/redis
- 6379/tcp
# BookStack (wiki.skudak.com) -- container publishes 6875:8080
- 6875/tcp
# Satisfactory
+3
View File
@@ -2,6 +2,9 @@
- import_tasks: firewall.yml
- import_tasks: podman/podman.yml
- import_tasks: podman/podman-prune.yml
tags: podman-prune
# WEB SERVER: Caddy is the default and only web server
# nginx has been completely replaced and removed
@@ -0,0 +1,36 @@
---
- name: template podman prune script
become: true
ansible.builtin.template:
src: podman-prune.sh.j2
dest: /usr/local/bin/podman-prune.sh
owner: root
group: root
mode: 0755
setype: bin_t
- name: template podman prune systemd service
become: true
ansible.builtin.template:
src: podman-prune.service.j2
dest: /etc/systemd/system/podman-prune.service
owner: root
group: root
mode: 0644
- name: template podman prune systemd timer
become: true
ansible.builtin.template:
src: podman-prune.timer.j2
dest: /etc/systemd/system/podman-prune.timer
owner: root
group: root
mode: 0644
- name: enable and start podman prune timer
become: true
ansible.builtin.systemd:
name: podman-prune.timer
enabled: true
state: started
daemon_reload: true