retire Graylog behind a flag, fix Caddy reloads, reap awsddns zombies
Graylog was the worst cost/benefit tenant on this 4-core box: two JVMs plus
MongoDB holding ~1.6 GB resident and ~3% CPU around the clock to store ~3k
messages a day -- about 28 MB across its four live indices. journald already
retains ~25 days of the same logs at its 500M cap, so this costs searchability,
not the logs.
The switch is `graylog_enabled` in inventory rather than a role default,
because three roles read it (common, podman, graylog-config). The disabled
path is an active teardown, not a skipped create: the containers already on
the host keep running and their systemd user units keep restarting them at
boot unless something stops and removes them. fluent-bit follows the same
flag -- with the GELF sink down it would spin retrying a dead 127.0.0.1:12202
and fill the journal it exists to drain -- but only the service state follows,
so re-enabling is a restart rather than a reinstall.
Caddy reloads were silently no-ops. The handler read /etc/caddy/Caddyfile,
which is a single-file bind mount, and podman binds those by inode; the
template module writes a temp file and renames it into place, so every deploy
gave the host file a new inode while the container kept seeing the one it was
created with. Config changes only ever landed when something recreated the
container. {{ caddy_path }}/config is also mounted, as a *directory*, and
directory mounts resolve names at open() time -- so /config/Caddyfile is
always the file Ansible just wrote.
awsddns and its four siblings had accumulated 12 zombies over 30 days of
uptime. The image's PID 1 is busybox crond, which only waitpid()s the job PIDs
it tracks and does no generic orphan reaping, so whenever the run-parts/sh
layer exited before the script it left a permanent <defunct>. init: true puts
catatonit at PID 1 to reap them, and the recreation clears the existing ones.
Also bumps fulfillr and greg-time-bot images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
co-authored by
Claude Opus 5
Claude Opus 5
parent
773a2bbc9c
commit
eea55def6c
@@ -11,11 +11,15 @@
|
||||
name: fail2ban
|
||||
state: restarted
|
||||
|
||||
# Guarded: a config change still notifies this handler when graylog_enabled is
|
||||
# false, and an unconditional restart would start the service the fluent-bit
|
||||
# task just stopped.
|
||||
- name: restart fluent-bit
|
||||
become: true
|
||||
ansible.builtin.systemd:
|
||||
name: fluent-bit
|
||||
state: restarted
|
||||
when: graylog_enabled | bool
|
||||
|
||||
- name: restart_journald
|
||||
become: true
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
---
|
||||
# Fluent Bit - Log forwarder from journald to Graylog GELF
|
||||
# Deployed as systemd service (not container) for direct journal access
|
||||
#
|
||||
# Graylog's GELF input is fluent-bit's only output, so the two share a switch:
|
||||
# with the stack down fluent-bit would just spin retrying a dead 127.0.0.1:12202
|
||||
# and filling the journal it is meant to be draining. The package and config
|
||||
# stay installed either way -- only the service follows graylog_enabled (see
|
||||
# inventories/home/hosts.yml) -- so re-enabling is a restart, not a reinstall.
|
||||
|
||||
- name: install fluent-bit package
|
||||
become: true
|
||||
@@ -37,9 +43,9 @@
|
||||
mode: '0644'
|
||||
notify: restart fluent-bit
|
||||
|
||||
- name: enable and start fluent-bit service
|
||||
- name: set fluent-bit service state to match graylog_enabled
|
||||
become: true
|
||||
ansible.builtin.systemd:
|
||||
name: fluent-bit
|
||||
enabled: true
|
||||
state: started
|
||||
enabled: "{{ graylog_enabled | bool }}"
|
||||
state: "{{ 'started' if (graylog_enabled | bool) else 'stopped' }}"
|
||||
|
||||
@@ -3,10 +3,19 @@
|
||||
# 1.9 TB root means it will happily grow into the tens of gigabytes; it had
|
||||
# reached 4 GB before this was set.
|
||||
#
|
||||
# Kept small on purpose: every container runs with log_driver=journald and
|
||||
# Every container runs with log_driver=journald, so this is where container
|
||||
# stdout lands.
|
||||
{% if graylog_enabled | bool %}
|
||||
# fluent-bit drains the journal into Graylog continuously (systemd input,
|
||||
# _COMM=conmon -- see templates/fluent-bit/fluent-bit.conf.j2), so Graylog is
|
||||
# the system of record. What stays here is only the buffer that covers
|
||||
# fluent-bit being down, and 500M is a long outage at this log rate.
|
||||
# the system of record and what stays here is only the buffer that covers
|
||||
# fluent-bit being down.
|
||||
{% else %}
|
||||
# Graylog is disabled (see graylog_enabled in inventories/home/hosts.yml), so
|
||||
# the journal is now the only log store. No increase was needed for that: the
|
||||
# cap is a size limit, not a time limit, and fluent-bit only ever read the
|
||||
# journal rather than rotating it, so retention is unchanged at roughly 25 days
|
||||
# at the current rate -- longer than Graylog's own four live indices covered.
|
||||
{% endif %}
|
||||
[Journal]
|
||||
SystemMaxUse={{ journald_max_use | default('500M') }}
|
||||
|
||||
@@ -19,10 +19,15 @@ Stages, in order — the ordering is deliberate, see the comments in the templat
|
||||
2. **SQLite snapshots** (`.backup`, then `pragma integrity_check`) where used.
|
||||
3. **rsync** of the data tree, config, and db dumps to TrueNAS.
|
||||
|
||||
Failures raise `status=failed` on the `nextcloud-backup` syslog tag, which an
|
||||
external Graylog rule matches to send mail. Do not rename that tag: it is shared
|
||||
by every instance including Gitea, and renaming it here silently stops alerting
|
||||
for all of them.
|
||||
Failures raise `status=failed` on the `nextcloud-backup` syslog tag. The mail
|
||||
itself is sent by the unit's own `OnFailure=` handler
|
||||
(`templates/nextcloud/nextcloud-backup-alert.sh.j2`) straight through
|
||||
`sendmail`, so alerting does **not** depend on Graylog and is unaffected by
|
||||
`graylog_enabled` being off. A Graylog rule matching the same tag is a
|
||||
secondary, dashboard-side copy of that signal.
|
||||
|
||||
Do not rename that tag: it is shared by every instance including Gitea, and
|
||||
renaming it here silently stops the Graylog-side alerting for all of them.
|
||||
|
||||
|
||||
## Restore
|
||||
@@ -141,3 +146,55 @@ LibreSign setup failures and buys nothing at this scale.
|
||||
- `LC_ALL` / `LANG` must be set or the JVM comes up as `ANSI_X3.4-1968` and
|
||||
LibreSign warns that accented characters in signer names will be mangled
|
||||
(LibreSign issue #4872).
|
||||
|
||||
|
||||
## Logging
|
||||
|
||||
Every container runs with `log_driver=journald`, so container stdout lands in
|
||||
the host journal, capped at 500M by `roles/common/templates/journald-size.conf.j2`
|
||||
(about 25 days at the current rate). `journalctl CONTAINER_NAME=<name>` is the
|
||||
day-to-day way to read it.
|
||||
|
||||
On top of that sits an optional Graylog stack — `graylog`, `graylog-opensearch`,
|
||||
`graylog-mongo`, fed by a host `fluent-bit` service that tails the journal and
|
||||
ships GELF to `127.0.0.1:12202`, enriched with the MaxMind GeoIP database, and
|
||||
configured over the REST API by the separate `graylog-config` role.
|
||||
|
||||
**It is off.** The switch is `graylog_enabled` in
|
||||
`inventories/home/hosts.yml`, and it gates all five of those pieces at once:
|
||||
|
||||
| `graylog_enabled` | effect |
|
||||
| --- | --- |
|
||||
| `true` | stack deployed, fluent-bit shipping, GeoIP downloaded, `graylog-config` runs, `logs.debyl.io` proxies to the UI and `/gelf` |
|
||||
| `false` | containers removed and their systemd user units disabled and deleted, fluent-bit stopped and disabled, GeoIP and `graylog-config` skipped, `logs.debyl.io` answers 503 |
|
||||
|
||||
It was turned off because the cost/benefit is bad on a 4-core box: two JVMs plus
|
||||
MongoDB held ~1.6 GB resident and ~3% of the CPU continuously to store roughly
|
||||
3k messages a day — about 28 MB of real log data across four live indices — all
|
||||
of which journald already keeps for longer.
|
||||
|
||||
Turning it back on is `graylog_enabled: true` plus `make deploy TAGS=graylog`.
|
||||
Nothing is destroyed by the off path: the volumes under `{{ graylog_path }}`
|
||||
keep the indices, the Mongo database holding streams/pipelines/dashboards, and
|
||||
the node-id file, so the stack comes back with its configuration intact.
|
||||
|
||||
Two things genuinely stop while it is off, both by design:
|
||||
|
||||
- **Search and dashboards.** The logs still exist in journald; the query
|
||||
interface over them does not.
|
||||
- **External GELF ingest.** The AWS Lambda that POSTs to `logs.debyl.io/gelf`
|
||||
for the `debyltech-api` stream gets a 503. Those events are dropped, not
|
||||
queued — nothing else records them.
|
||||
|
||||
### Caddy config reloads
|
||||
|
||||
The `reload caddy` handler deliberately reads `/config/Caddyfile`, not the
|
||||
`/etc/caddy/Caddyfile` the container starts from, even though both are the same
|
||||
host file. `/etc/caddy/Caddyfile` is a **single-file** bind mount, which podman
|
||||
binds by inode, and Ansible's `template` module writes a temp file and renames
|
||||
it into place — so every deploy gives the host file a new inode while the
|
||||
container keeps seeing the one it was created with. Reloading from that path
|
||||
silently re-applied the previous config; changes only landed when something
|
||||
recreated the container. `{{ caddy_path }}/config` is also bind-mounted as a
|
||||
*directory* at `/config`, and directory mounts resolve names at `open()` time,
|
||||
so `/config/Caddyfile` is always the file Ansible just wrote.
|
||||
|
||||
@@ -25,11 +25,21 @@
|
||||
tags:
|
||||
- caddy
|
||||
|
||||
# Reads /config/Caddyfile, NOT the /etc/caddy/Caddyfile the container starts
|
||||
# from, even though both are the same host file. /etc/caddy/Caddyfile is a
|
||||
# single-file bind mount, which podman binds by inode; the template module
|
||||
# writes a temp file and renames it into place, so every deploy gives the host
|
||||
# file a new inode and the container keeps seeing the one it was created with.
|
||||
# Reloading from that path silently re-applied the old config -- the change only
|
||||
# ever landed when something recreated the container. {{ caddy_path }}/config is
|
||||
# also bind-mounted as a *directory* at /config, and a directory mount resolves
|
||||
# names at open() time, so /config/Caddyfile is always the file Ansible just
|
||||
# wrote.
|
||||
- name: reload caddy
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
ansible.builtin.command: |
|
||||
podman exec caddy caddy reload --config /etc/caddy/Caddyfile
|
||||
podman exec caddy caddy reload --config /config/Caddyfile --adapter caddyfile
|
||||
tags:
|
||||
- caddy
|
||||
- caddy-config
|
||||
@@ -42,3 +52,12 @@
|
||||
scope: user
|
||||
tags:
|
||||
- zomboid
|
||||
|
||||
- name: reload podman systemd
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
scope: user
|
||||
tags:
|
||||
- podman
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
---
|
||||
# The image's PID 1 is busybox crond (ENTRYPOINT crond -f -d 8), which runs the
|
||||
# /etc/periodic/15min/awsddns script every quarter hour. busybox crond only
|
||||
# waitpid()s the job PIDs it is tracking -- it does no generic orphan reaping --
|
||||
# so whenever the run-parts/sh layer between crond and the script exits first,
|
||||
# the script is reparented to PID 1 and stays <defunct> forever. That had left
|
||||
# 12 zombies across these five containers after 30 days of uptime.
|
||||
#
|
||||
# init: true makes podman inject catatonit as PID 1 with crond as its child, and
|
||||
# catatonit reaps every orphan it inherits. Changing this recreates the
|
||||
# containers, which also clears the zombies already accumulated.
|
||||
- import_tasks: podman/podman-check.yml
|
||||
vars:
|
||||
container_name: awsddns
|
||||
@@ -13,6 +23,7 @@
|
||||
image: "{{ image }}"
|
||||
restart_policy: on-failure:3
|
||||
log_driver: journald
|
||||
init: true
|
||||
env:
|
||||
AWS_ZONE_TTL: 60
|
||||
AWS_ZONE_ID: "{{ aws_zone_id }}"
|
||||
@@ -40,6 +51,7 @@
|
||||
image: "{{ image }}"
|
||||
restart_policy: on-failure:3
|
||||
log_driver: journald
|
||||
init: true
|
||||
env:
|
||||
AWS_ZONE_TTL: 60
|
||||
AWS_ZONE_ID: "{{ aws_skudak_zone_id }}"
|
||||
@@ -67,6 +79,7 @@
|
||||
image: "{{ image }}"
|
||||
restart_policy: on-failure:3
|
||||
log_driver: journald
|
||||
init: true
|
||||
env:
|
||||
AWS_ZONE_TTL: 60
|
||||
AWS_ZONE_ID: "{{ fulfillr_zone_id }}"
|
||||
@@ -96,6 +109,7 @@
|
||||
image: "{{ image }}"
|
||||
restart_policy: on-failure:3
|
||||
log_driver: journald
|
||||
init: true
|
||||
env:
|
||||
AWS_ZONE_TTL: 60
|
||||
AWS_ZONE_ID: "{{ fulfillr_zone_id }}"
|
||||
@@ -123,6 +137,7 @@
|
||||
image: "{{ image }}"
|
||||
restart_policy: on-failure:3
|
||||
log_driver: journald
|
||||
init: true
|
||||
env:
|
||||
AWS_ZONE_TTL: 60
|
||||
AWS_ZONE_ID: "Z07501202A6AYMHCVP50A"
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
# Runs in place of graylog.yml when graylog_enabled is false.
|
||||
#
|
||||
# A bare `when:` on the import would only stop Ansible from *creating* the
|
||||
# stack; containers already on the host would keep running and their systemd
|
||||
# user units would keep starting them at boot. So the disabled path has to be an
|
||||
# active teardown: disable the units, then remove the containers.
|
||||
#
|
||||
# Volumes under {{ graylog_path }} are deliberately untouched -- indices, the
|
||||
# Mongo database holding streams/pipelines/dashboards, and the node-id file all
|
||||
# survive, so flipping graylog_enabled back to true resumes where this left off.
|
||||
|
||||
# graylog is stopped first: its unit `requires` the other two, and tearing down
|
||||
# a dependency out from under it makes the shutdown noisy for no reason.
|
||||
- name: stop and disable graylog stack systemd units
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
ansible.builtin.systemd:
|
||||
name: "{{ item }}.service"
|
||||
enabled: false
|
||||
state: stopped
|
||||
daemon_reload: true
|
||||
scope: user
|
||||
loop:
|
||||
- graylog
|
||||
- graylog-opensearch
|
||||
- graylog-mongo
|
||||
# Left over from the Elasticsearch-to-OpenSearch migration; the container is
|
||||
# long gone but the enabled unit still tries to start it every boot.
|
||||
- graylog-elastic
|
||||
register: graylog_units
|
||||
failed_when: false
|
||||
tags: graylog
|
||||
|
||||
- name: remove graylog stack containers
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
containers.podman.podman_container:
|
||||
name: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- graylog
|
||||
- graylog-opensearch
|
||||
- graylog-mongo
|
||||
tags: graylog
|
||||
|
||||
- name: remove graylog stack systemd unit files
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
ansible.builtin.file:
|
||||
path: "{{ podman_home }}/.config/systemd/user/{{ item }}.service"
|
||||
state: absent
|
||||
loop:
|
||||
- graylog
|
||||
- graylog-opensearch
|
||||
- graylog-mongo
|
||||
- graylog-elastic
|
||||
notify: reload podman systemd
|
||||
tags: graylog
|
||||
@@ -81,13 +81,13 @@
|
||||
|
||||
- import_tasks: containers/debyltech/fulfillr.yml
|
||||
vars:
|
||||
image: git.debyl.io/debyltech/fulfillr:20260728.2155
|
||||
image: git.debyl.io/debyltech/fulfillr:20260825.1909
|
||||
tags: debyltech, fulfillr
|
||||
|
||||
# Staging back-office (fulfillr-dev.debyltech.com) — same image, staging Turso config.
|
||||
- import_tasks: containers/debyltech/fulfillr-dev.yml
|
||||
vars:
|
||||
image: git.debyl.io/debyltech/fulfillr:20260728.2155
|
||||
image: git.debyl.io/debyltech/fulfillr:20260825.1909
|
||||
tags: debyltech, fulfillr-dev
|
||||
|
||||
- import_tasks: containers/debyltech/uptime-kuma.yml
|
||||
@@ -100,7 +100,10 @@
|
||||
image: docker.io/louislam/uptime-kuma:2.3.2
|
||||
tags: home, uptime
|
||||
|
||||
# GeoIP is only consumed by Graylog's enrichment pipelines, so it follows the
|
||||
# same switch -- see graylog_enabled in inventories/home/hosts.yml.
|
||||
- import_tasks: data/geoip.yml
|
||||
when: graylog_enabled | bool
|
||||
tags: graylog, geoip
|
||||
|
||||
- import_tasks: containers/debyltech/graylog.yml
|
||||
@@ -108,16 +111,26 @@
|
||||
mongo_image: docker.io/mongo:7.0
|
||||
opensearch_image: docker.io/opensearchproject/opensearch:2
|
||||
image: docker.io/graylog/graylog:7.0.1
|
||||
when: graylog_enabled | bool
|
||||
tags: debyltech, graylog
|
||||
|
||||
# The disabled path is an active teardown, not just a skipped create: without it
|
||||
# the containers already on the host keep running and their systemd user units
|
||||
# keep restarting them at boot.
|
||||
- import_tasks: containers/debyltech/graylog-teardown.yml
|
||||
when: not (graylog_enabled | bool)
|
||||
tags: debyltech, graylog
|
||||
|
||||
- import_tasks: containers/home/gregtime.yml
|
||||
vars:
|
||||
image: localhost/greg-time-bot:3.10.0
|
||||
image: localhost/greg-time-bot:3.14.1
|
||||
tags: gregtime
|
||||
|
||||
# Gated off by default — see zomboid_enabled in roles/podman/defaults/main.yml.
|
||||
- import_tasks: containers/home/zomboid.yml
|
||||
vars:
|
||||
image: docker.io/cm2network/steamcmd:root
|
||||
when: zomboid_enabled | bool
|
||||
tags: zomboid
|
||||
|
||||
# ---------------------------------------------------------- Gitea backups
|
||||
|
||||
@@ -238,6 +238,34 @@
|
||||
}
|
||||
|
||||
# Graylog Logs - {{ logs_server_name }}
|
||||
{% if not (graylog_enabled | bool) %}
|
||||
# Graylog is disabled (see graylog_enabled in inventories/home/hosts.yml). The
|
||||
# vhost is kept rather than dropped so the name keeps resolving and its
|
||||
# certificate keeps renewing; both handlers answer 503 instead of proxying to a
|
||||
# port nothing is listening on, which would hang and then 502.
|
||||
#
|
||||
# NOTE: this means the external Lambda that POSTs to /gelf for the
|
||||
# `debyltech-api` stream is being turned away. Its events are dropped, not
|
||||
# queued -- nothing else records them.
|
||||
{{ logs_server_name }} {
|
||||
handle /gelf {
|
||||
respond "Log ingest disabled" 503
|
||||
}
|
||||
|
||||
handle {
|
||||
respond "Graylog is disabled" 503
|
||||
}
|
||||
|
||||
log {
|
||||
output file /var/log/caddy/graylog.log {
|
||||
roll_size {{ caddy_log_roll_size }}
|
||||
roll_keep {{ caddy_log_roll_keep }}
|
||||
roll_keep_for {{ caddy_log_roll_keep_for }}
|
||||
}
|
||||
format json
|
||||
}
|
||||
}
|
||||
{% else %}
|
||||
{{ logs_server_name }} {
|
||||
# GELF HTTP endpoint - open for Lambda (auth via header)
|
||||
# Must come BEFORE ip_restricted_site to allow external access
|
||||
@@ -283,6 +311,7 @@
|
||||
format json
|
||||
}
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
# ============================================================================
|
||||
# COMPLEX CONFIGURATIONS
|
||||
|
||||
Reference in New Issue
Block a user