feat(gitea-actions): build the CI job images in Gitea CI

The runner's job images were built by ansible into localhost/ only, so the
nightly CI prune deleted them and every idle stretch ended with CI failing in
under a second on `docker pull localhost/gitea-ci:latest` until someone re-ran
the role and waited out a rebuild. The previous commit moved them to the Gitea
registry; this moves the *build* off the deploy path entirely.

- .gitea/workflows/ci-images.yml builds files/Containerfile.* and pushes to
  git.debyl.io/gitbot/. Per-image change detection, so an ESP-IDF pin bump does
  not rebuild the other two; weekly schedule for base-image updates; a
  workflow_dispatch selector. PRs build under a throwaway :pr-<n> tag and drop
  it -- the build lands in the live runner's store, and act_runner will not
  re-pull a tag it already has, so a PR using the real tag would hand every
  later job on this host an unmerged image.
- The Containerfiles stop being ansible templates: their version vars are now
  --build-arg, read by the workflow out of the same defaults/main.yml the role
  interpolates, so CI and ansible build the same bytes from one set of pins.
- LABEL io.debyl.ci-base moves into each Containerfile so neither builder can
  forget the prune exemption; the workflow re-checks it before pushing.
- roles/gitea-actions pulls instead of building. gitea_ci_build_local=true
  restores the local build+push for seeding a cold registry or when CI is
  down -- the workflow that builds gitea-ci runs in gitea-ci.
- Lint .gitea/ alongside ansible/, and document the flow in the role README.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ansible/roles/gitea-actions/README.md

@@ -0,0 +1,91 @@
+# gitea-actions
+
+Runs the Gitea Actions runners on `home.debyl.io`. One `act_runner` process per
+Gitea instance (`git.debyl.io`, `git.skudak.com`), both as the `gitea-runner`
+user, both backed by the same rootless podman image store.
+
+## CI job images
+
+Jobs do not run on the host. Each one gets an ephemeral container from one of
+three images:
+
+| `runs-on` / `container:` | Image | Used by |
+| --- | --- | --- |
+| `fedora`, `ubuntu-latest`, `ubuntu-22.04` | `git.debyl.io/gitbot/gitea-ci:latest` | Go / node / web jobs, `docker build` |
+| `container: image:` | `git.debyl.io/gitbot/gitea-ci-espidf:<esp_idf_version>` | esp-mg-tpms, skudak/esp32-stm32-vcu |
+| `container: image:` | `git.debyl.io/gitbot/gitea-ci-platformio:<pio_espressif32_version>` | skudak/esp32-web-interface |
+
+**This role does not build them.** `.gitea/workflows/ci-images.yml` builds
+`files/Containerfile.*` and pushes to the Gitea registry; the role logs
+`gitea-runner` in and pulls. Version pins live in `defaults/main.yml` and are
+read by both the role and the workflow, so a bump moves the image tag in one
+place.
+
+### Why the registry
+
+The images used to exist only as `localhost/gitea-ci*` in the runner's store.
+The nightly prune (`roles/podman`, `podman_prune_ci_until: 48h`) deletes any
+CI-user image older than that which no container holds, so after an idle
+weekend every job failed in under a second on `docker pull
+localhost/gitea-ci:latest`, and the only fix was re-running this role and
+waiting out a full rebuild.
+
+Two things now keep that from happening:
+
+- **A registry copy.** `force_pull` stays `false`, which in act_runner means
+  *pull only when missing* — so a present image is never re-fetched, and a
+  pruned one is restored by the next job without anyone noticing.
+- **A prune exemption.** Each Containerfile declares
+  `LABEL io.debyl.ci-base="true"`, and the prune skips that label
+  (`podman_prune_ci_keep_label`). Its `until` counts from build time, not pull
+  time, so without this a re-pulled image would be deleted again the same night
+  — a 7.8 GB ESP-IDF download every single day.
+
+The label is declared in the Containerfile rather than passed as `--label` so
+neither builder can omit it; the workflow re-checks it with `docker inspect`
+before pushing.
+
+### Authentication
+
+Both the role and act_runner read `/home/gitea-runner/.docker/config.json`.
+act_runner uses it for the job-image pull it performs when a label's image is
+missing; podman falls back to the same file. The role writes it from
+`gitea_registry_username` / `gitea_registry_token` (vault), so one login covers
+both. The `skudak` runner pulls from `git.debyl.io` too — same host, same user,
+same file.
+
+The workflow pushes with a `REGISTRY_TOKEN` secret on `bastian/deploy_home`,
+belonging to the same `gitbot` user: Gitea authorises a package push by the
+token's owner, not by the path, so pushing to `gitbot/` means logging in as
+`gitbot`.
+
+### Rebuilding
+
+Normally nothing to do — edit a `files/Containerfile.*` or a version pin, push
+to `master`, and the workflow rebuilds only the affected images. It also
+rebuilds everything weekly so base-image updates land without a commit, and
+takes a `workflow_dispatch` with an image selector.
+
+Pull requests build but do not push, under a throwaway `:pr-<n>` tag that is
+deleted afterwards. The build runs in the live runner's image store, so a PR
+tagged with the real name would hand every later job on this host an unmerged
+image.
+
+### Bootstrap / CI is down
+
+The workflow that builds `gitea-ci` runs *in* `gitea-ci`, so a registry that has
+never held it cannot bootstrap itself. Build on the host instead:
+
+```sh
+make deploy TAGS=gitea-actions EXTRA_VARS="gitea_ci_build_local=true"
+```
+
+That builds all three from the same Containerfiles and pushes them. One run is
+enough even on a cold registry: `tasks/main.yml` imports `images.yml` before
+`runner.yml`, so the images are published before the runner labels are flipped
+to point at them.
+
+The alternative first-time path is to merge the workflow and dispatch it while
+the deployed labels still say `localhost/` — the job then builds inside the old
+local image and seeds the registry — then run a plain
+`make deploy TAGS=gitea-actions` to switch the labels over.

ansible/roles/gitea-actions/defaults/main.yml

@@ -22,17 +22,18 @@ act_runner_bin: /usr/local/bin/act_runner
 act_runner_config_dir: /etc/act_runner
 act_runner_work_dir: /var/lib/act_runner
 
-# Job container images. tasks/images.yml builds them into the gitea-runner
-# rootless store and pushes them to the Gitea container registry.
+# Job container images, served from the Gitea container registry.
 #
-# They used to live only under localhost/, and the nightly podman prune
-# (roles/podman: podman_prune_ci_until) deletes any CI-user image older than
-# 48h that no container is using -- so every idle weekend CI failed in 0-1s on
-# `docker pull localhost/gitea-ci:latest` until someone re-ran this role and
-# waited out a full rebuild. With a registry copy, a pruned image is simply
-# re-pulled by the next job (force_pull stays false, so a present image is
-# never re-pulled), and this role pulls instead of rebuilding when the
-# Containerfile has not changed.
+# They used to live only under localhost/, built by this role. The nightly
+# podman prune (roles/podman: podman_prune_ci_until) deletes any CI-user image
+# older than 48h that no container is using, so every idle weekend CI failed in
+# 0-1s on `docker pull localhost/gitea-ci:latest` until someone re-ran the role
+# and waited out a full rebuild.
+#
+# Now .gitea/workflows/ci-images.yml builds them from files/Containerfile.* and
+# pushes them here, and this role only pulls. A pruned image is re-pulled by the
+# next job on its own (force_pull stays false, which means "pull only when
+# missing", so a present image is never re-fetched).
 #
 # Workflows that pin `container: image:` must use these registry paths too
 # (esp-mg-tpms, skudak/esp32-stm32-vcu, skudak/esp32-web-interface).
@@ -55,26 +56,36 @@ gitea_ci_platformio_image: "{{ gitea_ci_registry }}/{{ gitea_ci_registry_namespa
 # fallback auth file), so one login covers the runner and this role.
 gitea_ci_registry_authfile: "{{ gitea_runner_home }}/.docker/config.json"
 
-# Label stamped on the CI base images so the nightly prune skips them; must match
-# podman_prune_ci_keep_label in roles/podman/defaults/main.yml. Without it the
-# prune (whose `until` counts from build time, not pull time) would delete a
-# re-pulled image again the next night -- a 7.8 GB ESP-IDF re-download after
-# every idle day. Superseded tags (e.g. after an esp_idf_version bump) are
-# therefore kept too; remove them by hand.
-gitea_ci_keep_label: io.debyl.ci-base
-
+# The images this role keeps present on the runner. `build_args` is a literal
+# podman-build argument string (podman_image has no structured build-arg
+# option) and is only used by the gitea_ci_build_local fallback below -- the
+# workflow passes the same --build-arg values, read out of the version vars
+# above, so there is one source of truth for the pins.
 gitea_ci_images:
   - image: "{{ gitea_ci_image }}"
     containerfile: Containerfile.ci
-    template: Containerfile.ci
+    build_args: ""
   - image: "{{ gitea_ci_espidf_image }}"
     containerfile: Containerfile.espidf
-    template: Containerfile.espidf.j2
+    build_args: "--build-arg ESP_IDF_VERSION={{ esp_idf_version }}"
   - image: "{{ gitea_ci_platformio_image }}"
     containerfile: Containerfile.platformio
-    template: Containerfile.platformio.j2
+    build_args: >-
+      --build-arg PLATFORMIO_CORE_VERSION={{ platformio_core_version }}
+      --build-arg PIO_ESPRESSIF32_VERSION={{ pio_espressif32_version }}
 
-# Default labels for every runner — map runs-on values to the local CI image.
+# Escape hatch: build the images on the host and push them, instead of pulling
+# what CI published. Needed to seed a brand-new registry namespace, and when CI
+# itself is down -- the workflow that builds gitea-ci runs *in* gitea-ci, so a
+# registry that has never held it cannot bootstrap itself.
+#
+#   make deploy TAGS=gitea-actions EXTRA_VARS="gitea_ci_build_local=true"
+#
+# Off by default: a plain deploy should never sit through a 15-minute ESP-IDF
+# rebuild, and two publishers racing on the same tag is worth avoiding.
+gitea_ci_build_local: false
+
+# Default labels for every runner — map runs-on values to the registry CI image.
 # Firmware jobs opt into the ESP-IDF image per-job via `container:` in their workflow.
 gitea_runner_labels:
   - "fedora:docker://{{ gitea_ci_image }}"

ansible/roles/gitea-actions/files/Containerfile.ci

@@ -1,8 +1,18 @@
 # Default Gitea Actions job image (managed by ansible: roles/gitea-actions).
 # Covers Go/web/node jobs plus `docker build` (talks to the mounted rootless
 # podman socket). Go toolchains are provided per-job by actions/setup-go.
+#
+# Built and published by .gitea/workflows/ci-images.yml; roles/gitea-actions
+# only pulls the result (see gitea_ci_build_local for the local-build fallback).
+# A plain Containerfile, not a template, so CI and ansible build the same bytes.
 FROM node:20-bookworm-slim
 
+# Exempts the image from the nightly CI prune -- see podman_prune_ci_keep_label
+# in roles/podman/defaults/main.yml. Declared here rather than passed as a
+# --label at build time so neither builder can forget it: without the label the
+# prune deletes the image every night and the next job re-pulls a gigabyte.
+LABEL io.debyl.ci-base="true"
+
 ARG DOCKER_CLI_VERSION=27.3.1
 
 RUN apt-get update && apt-get install -y --no-install-recommends \

ansible/roles/gitea-actions/files/Containerfile.espidf

@@ -14,7 +14,19 @@
 # the release aborts *after* the firmware and version.json are already live —
 # clients get the new build while the tag, Gitea release and protocol manifest
 # are never written. Keep it installed.
-FROM espressif/idf:{{ esp_idf_version }}
+#
+# Built and published by .gitea/workflows/ci-images.yml; roles/gitea-actions
+# only pulls the result. ESP_IDF_VERSION is a build arg rather than an ansible
+# template var so CI and ansible build the same bytes -- its value is read from
+# esp_idf_version in roles/gitea-actions/defaults/main.yml by both.
+ARG ESP_IDF_VERSION
+FROM espressif/idf:${ESP_IDF_VERSION}
+
+# Exempts the image from the nightly CI prune -- see podman_prune_ci_keep_label
+# in roles/podman/defaults/main.yml. Declared here rather than passed as a
+# --label at build time so neither builder can forget it: without the label the
+# prune deletes the image every night and the next job re-pulls 7.8 GB.
+LABEL io.debyl.ci-base="true"
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
         curl ca-certificates unzip jq python3-yaml python3-jinja2 \

ansible/roles/gitea-actions/files/Containerfile.platformio

@@ -8,8 +8,23 @@
 # was validated on hardware — bump pio_espressif32_version /
 # platformio_core_version in defaults/main.yml to upgrade (the image tag
 # tracks the platform version).
+#
+# Built and published by .gitea/workflows/ci-images.yml; roles/gitea-actions
+# only pulls the result. The pins are build args rather than ansible template
+# vars so CI and ansible build the same bytes -- their values are read from
+# platformio_core_version / pio_espressif32_version in
+# roles/gitea-actions/defaults/main.yml by both.
 FROM python:3.12-slim-bookworm
 
+ARG PLATFORMIO_CORE_VERSION
+ARG PIO_ESPRESSIF32_VERSION
+
+# Exempts the image from the nightly CI prune -- see podman_prune_ci_keep_label
+# in roles/podman/defaults/main.yml. Declared here rather than passed as a
+# --label at build time so neither builder can forget it: without the label the
+# prune deletes the image every night and the next job re-pulls a gigabyte.
+LABEL io.debyl.ci-base="true"
+
 ENV PLATFORMIO_CORE_DIR=/opt/platformio
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -18,12 +33,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && apt-get install -y --no-install-recommends nodejs \
     && rm -rf /var/lib/apt/lists/*
 
-RUN pip install --no-cache-dir platformio=={{ platformio_core_version }}
+RUN pip install --no-cache-dir platformio==${PLATFORMIO_CORE_VERSION}
 
 # Seed project mirroring the real projects' platformio.ini so `pio pkg install`
 # pulls the platform + toolchain + framework packages into the core dir.
+# %s + a quoted argument, not ${...} inside the single-quoted format string:
+# RUN is `sh -c`, and sh does not expand inside single quotes, so an inlined
+# ${PIO_ESPRESSIF32_VERSION} would be written to platformio.ini literally.
 RUN mkdir -p /tmp/seed/src \
-    && printf '[env:seed]\nplatform = espressif32@{{ pio_espressif32_version }}\nframework = arduino\nboard = esp32dev\nboard_build.filesystem = spiffs\nplatform_packages = platformio/tool-esptoolpy\n' > /tmp/seed/platformio.ini \
+    && printf '[env:seed]\nplatform = espressif32@%s\nframework = arduino\nboard = esp32dev\nboard_build.filesystem = spiffs\nplatform_packages = platformio/tool-esptoolpy\n' "${PIO_ESPRESSIF32_VERSION}" > /tmp/seed/platformio.ini \
     && pio pkg install -d /tmp/seed \
     && pio pkg install -d /tmp/seed --tool platformio/tool-mkspiffs \
     && rm -rf /tmp/seed \

ansible/roles/gitea-actions/tasks/images.yml

@@ -1,17 +1,10 @@
 ---
-# CI job images: stage each Containerfile, restore the image from the Gitea
-# registry if the nightly prune removed it, rebuild only when the Containerfile
-# changed, then push so the registry always holds what the runner uses.
-# See gitea_ci_images in defaults/main.yml.
-- name: create CI image build directory
-  become: true
-  become_user: "{{ gitea_runner_user }}"
-  ansible.builtin.file:
-    path: "{{ gitea_runner_home }}/ci-images"
-    state: directory
-    mode: "0755"
-  tags: gitea-actions
-
+# CI job images. .gitea/workflows/ci-images.yml builds files/Containerfile.*
+# and pushes them to the Gitea registry; this role only logs the runner in and
+# makes sure the images are present, so a plain deploy never waits on a build.
+#
+# Set gitea_ci_build_local=true to build and push from here instead -- see the
+# comment on that variable in defaults/main.yml.
 - name: create gitea-runner registry auth directory
   become: true
   become_user: "{{ gitea_runner_user }}"
@@ -21,6 +14,9 @@
     mode: "0700"
   tags: gitea-actions
 
+# Docker-format path on purpose: act_runner reads ~/.docker/config.json to
+# authenticate the job-image pull it does when a label's image is missing, and
+# podman falls back to the same file. One login covers both.
 - name: log gitea-runner in to the Gitea container registry
   become: true
   become_user: "{{ gitea_runner_user }}"
@@ -34,22 +30,7 @@
   no_log: true
   tags: gitea-actions
 
-- name: stage CI Containerfiles
-  become: true
-  become_user: "{{ gitea_runner_user }}"
-  ansible.builtin.template:
-    src: "{{ item.template }}"
-    dest: "{{ gitea_runner_home }}/ci-images/{{ item.containerfile }}"
-    mode: "0644"
-  loop: "{{ gitea_ci_images }}"
-  loop_control:
-    label: "{{ item.containerfile }}"
-  register: ci_containerfiles
-  tags: gitea-actions
-
-# A missing image here is normal (first push, or a new tag): the build below
-# creates it. Anything already present locally is left untouched.
-- name: restore CI images from the registry
+- name: pull CI images from the registry
   become: true
   become_user: "{{ gitea_runner_user }}"
   containers.podman.podman_image:
@@ -60,8 +41,35 @@
   loop: "{{ gitea_ci_images }}"
   loop_control:
     label: "{{ item.image }}"
-  when: not (ci_containerfiles.results | selectattr('item.image', 'equalto', item.image) | first).changed
-  failed_when: false
+  when: not (gitea_ci_build_local | bool)
+  tags: gitea-actions
+
+# --- local build fallback (gitea_ci_build_local=true) ------------------------
+# Only reached when seeding a new namespace or when CI cannot build for us.
+- name: create CI image build directory
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.file:
+    path: "{{ gitea_runner_home }}/ci-images"
+    state: directory
+    mode: "0755"
+  when: gitea_ci_build_local | bool
+  tags: gitea-actions
+
+# copy, not template: these are plain Containerfiles that CI builds verbatim.
+# Versions come in as --build-arg from the same defaults/main.yml the workflow
+# reads, so neither builder can drift from the other.
+- name: stage CI Containerfiles
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.copy:
+    src: "{{ item.containerfile }}"
+    dest: "{{ gitea_runner_home }}/ci-images/{{ item.containerfile }}"
+    mode: "0644"
+  loop: "{{ gitea_ci_images }}"
+  loop_control:
+    label: "{{ item.containerfile }}"
+  when: gitea_ci_build_local | bool
   tags: gitea-actions
 
 - name: build and push CI images
@@ -72,9 +80,8 @@
     path: "{{ gitea_runner_home }}/ci-images"
     build:
       file: "{{ gitea_runner_home }}/ci-images/{{ item.containerfile }}"
-      # Exempts the image from the nightly CI prune (gitea_ci_keep_label).
-      extra_args: "--label {{ gitea_ci_keep_label }}=true"
-    force: "{{ (ci_containerfiles.results | selectattr('item.image', 'equalto', item.image) | first).changed }}"
+      extra_args: "{{ item.build_args }}"
+    force: true
     push: true
     auth_file: "{{ gitea_ci_registry_authfile }}"
   environment:
@@ -82,4 +89,5 @@
   loop: "{{ gitea_ci_images }}"
   loop_control:
     label: "{{ item.image }}"
+  when: gitea_ci_build_local | bool
   tags: gitea-actions

ansible/roles/podman/defaults/main.yml

@@ -311,8 +311,13 @@ podman_prune_ci_users:
   - gitea-runner
   - actions-runner
 podman_prune_ci_until: 48h
-# CI base images built by roles/gitea-actions carry this label (gitea_ci_keep_label
-# there -- keep the two in sync) and are skipped by the CI image prune.
+# The CI base images declare LABEL io.debyl.ci-base="true" in
+# roles/gitea-actions/files/Containerfile.* (and .gitea/workflows/ci-images.yml
+# verifies it before publishing -- keep all three in sync). Images carrying it
+# are skipped below: `until` counts from build time and not pull time, so
+# without the exemption a re-pulled image would be deleted again the next
+# night, a 7.8 GB ESP-IDF re-download after every idle day. Superseded tags
+# (e.g. after an esp_idf_version bump) survive too; remove those by hand.
 podman_prune_ci_keep_label: io.debyl.ci-base
 
 # Daily rather than weekly: CI turns over many images a day, and a week of that

ansible/roles/podman/tasks/main.yml

@@ -123,7 +123,7 @@
 
 - import_tasks: containers/home/gregtime.yml
   vars:
-    image: localhost/greg-time-bot:3.18.1
+    image: localhost/greg-time-bot:3.18.3
   tags: gregtime
 
 # Built and loaded by `make deploy-remote` in ~/src/rsvp-debylio; bump this to