SCRUM-97: Healthcheck + restart-on-unhealthy for fulfillr containers

After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.

- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
  the alpine image), interval 30s / timeout 5s / retries 3 /
  start_period 30s (covers the ~14s HMAC retry backoff), failure_action
  restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
  build carrying the /health/startup probe).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
2026-06-28 15:35:22 -04:00
parent 511db704a4
commit 5d0d15f414
3 changed files with 25 additions and 3 deletions
@@ -55,6 +55,17 @@
image_strict: true image_strict: true
command: --config /config/dev.json command: --config /config/dev.json
restart_policy: on-failure:3 restart_policy: on-failure:3
# Self-heal the SCRUM-97 boot race: if a transient Secrets-Manager blip left the
# gated outreach/newsletter/cases routes unregistered, /health/startup returns 503
# and podman restarts the container in place (busybox wget ships in the alpine
# image; start_period covers the ~14s HMAC retry backoff so a healthy boot is never
# flagged).
healthcheck: "wget -q -O /dev/null http://localhost:8080/api/v1/health/startup || exit 1"
healthcheck_interval: 30s
healthcheck_timeout: 5s
healthcheck_retries: 3
healthcheck_start_period: 30s
healthcheck_failure_action: restart
log_driver: journald log_driver: journald
volumes: volumes:
- "{{ fulfillr_dev_path }}:/config" - "{{ fulfillr_dev_path }}:/config"
@@ -52,6 +52,17 @@
image_strict: true image_strict: true
command: --config /config/production.json command: --config /config/production.json
restart_policy: on-failure:3 restart_policy: on-failure:3
# Self-heal the SCRUM-97 boot race: if a transient Secrets-Manager blip left the
# gated outreach/newsletter/cases routes unregistered, /health/startup returns 503
# and podman restarts the container in place (busybox wget ships in the alpine
# image; start_period covers the ~14s HMAC retry backoff so a healthy boot is never
# flagged).
healthcheck: "wget -q -O /dev/null http://localhost:8080/api/v1/health/startup || exit 1"
healthcheck_interval: 30s
healthcheck_timeout: 5s
healthcheck_retries: 3
healthcheck_start_period: 30s
healthcheck_failure_action: restart
log_driver: journald log_driver: journald
volumes: volumes:
- "{{ fulfillr_path }}:/config" - "{{ fulfillr_path }}:/config"
+3 -3
View File
@@ -78,13 +78,13 @@
- import_tasks: containers/debyltech/fulfillr.yml - import_tasks: containers/debyltech/fulfillr.yml
vars: vars:
image: git.debyl.io/debyltech/fulfillr:20260614.1925 image: git.debyl.io/debyltech/fulfillr:20260628.1930
tags: debyltech, fulfillr tags: debyltech, fulfillr
# Staging back-office (fulfillr-dev.debyltech.com) — same image, staging Turso config. # Staging back-office (fulfillr-dev.debyltech.com) — same image, staging Turso config.
- import_tasks: containers/debyltech/fulfillr-dev.yml - import_tasks: containers/debyltech/fulfillr-dev.yml
vars: vars:
image: git.debyl.io/debyltech/fulfillr:20260614.1925 image: git.debyl.io/debyltech/fulfillr:20260628.1930
tags: debyltech, fulfillr-dev tags: debyltech, fulfillr-dev
- import_tasks: containers/debyltech/uptime-kuma.yml - import_tasks: containers/debyltech/uptime-kuma.yml
@@ -109,7 +109,7 @@
- import_tasks: containers/home/gregtime.yml - import_tasks: containers/home/gregtime.yml
vars: vars:
image: localhost/greg-time-bot:3.9.6 image: localhost/greg-time-bot:3.9.14
tags: gregtime tags: gregtime
- import_tasks: containers/home/zomboid.yml - import_tasks: containers/home/zomboid.yml