SCRUM-97: Healthcheck + restart-on-unhealthy for fulfillr containers #7

2026-06-28T15:35:42-04:00

bastian commented

2026-06-28 15:35:42 -04:00

Self-heals the power-cycle failure where go-fulfillr's gated routes (newsletter/outreach/cases) 404 because a transient boot-time HMAC Secrets Manager fetch left them unregistered, with the process still up so nothing restarts it.

fulfillr.yml + fulfillr-dev.yml: add a podman healthcheck on the containers.podman.podman_container task — wget the new dependency-free /api/v1/health/startup probe (busybox wget ships in the alpine image), interval 30s / timeout 5s / retries 3 / start_period 30s (covers the ~14s HMAC retry backoff so a healthy boot is never flagged), and healthcheck_failure_action: restart (--health-on-failure=restart) so podman restarts the container in place when those routes failed to register. Existing restart_policy: on-failure:3 kept (process-exit case).
main.yml: bump fulfillr + fulfillr-dev image → 20260628.1930 (the go-fulfillr build that adds /health/startup, SCRUM-97 PR #21, merged).

Deploy after merge: make deploy TAGS=fulfillr-dev (verify podman inspect ... Health.Status=healthy), then make deploy TAGS=fulfillr.

🤖 Generated with Claude Code

Self-heals the power-cycle failure where go-fulfillr's gated routes (newsletter/outreach/cases) 404 because a transient boot-time HMAC Secrets Manager fetch left them unregistered, with the process still up so nothing restarts it. - `fulfillr.yml` + `fulfillr-dev.yml`: add a podman healthcheck on the `containers.podman.podman_container` task — `wget` the new dependency-free `/api/v1/health/startup` probe (busybox wget ships in the alpine image), `interval 30s / timeout 5s / retries 3 / start_period 30s` (covers the ~14s HMAC retry backoff so a healthy boot is never flagged), and **`healthcheck_failure_action: restart`** (`--health-on-failure=restart`) so podman restarts the container in place when those routes failed to register. Existing `restart_policy: on-failure:3` kept (process-exit case). - `main.yml`: bump `fulfillr` + `fulfillr-dev` image → `20260628.1930` (the go-fulfillr build that adds `/health/startup`, SCRUM-97 PR #21, merged). Deploy after merge: `make deploy TAGS=fulfillr-dev` (verify `podman inspect ... Health.Status=healthy`), then `make deploy TAGS=fulfillr`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bastian added 1 commit 2026-06-28 15:35:43 -04:00

SCRUM-97: Healthcheck + restart-on-unhealthy for fulfillr containers 5d0d15f414

After a power cycle a transient HMAC Secrets Manager blip leaves
go-fulfillr's gated routes unregistered (404) with the process still up,
so nothing restarts it. Add a podman healthcheck probing the new
dependency-free /api/v1/health/startup (503 until those routes register)
with healthcheck_failure_action: restart, so podman restarts the
container in place and the next boot self-heals.

- fulfillr.yml + fulfillr-dev.yml: healthcheck via busybox wget (ships in
  the alpine image), interval 30s / timeout 5s / retries 3 /
  start_period 30s (covers the ~14s HMAC retry backoff), failure_action
  restart. Existing restart_policy on-failure:3 kept (process-exit case).
- main.yml: bump fulfillr + fulfillr-dev image to 20260628.1930 (the
  build carrying the /health/startup probe).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bastian merged commit 8a5ebb1559 into master

2026-06-28 15:35:51 -04:00

bastian deleted branch scrum-97/fulfillr-healthcheck

2026-06-28 15:35:51 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: bastian/deploy_home#7