gitea-actions: run CI jobs in rootless-podman containers

Switch the act_runners from :host execution to docker:// images backed by
a rootless podman socket under the gitea-runner user, so each job runs in
its own ephemeral container with per-job Go caches. This eliminates the
cross-repo GOMODCACHE/go-build poisoning that forced the debyl runner to
capacity:1.

- deps.yml: enable the rootless --user podman.socket, ensure subuid/subgid,
  register gitea_runner_uid; drop the rootful system socket override,
  podman-docker and host golang
- images.yml + Containerfile.ci/.espidf: build localhost/gitea-ci and
  localhost/gitea-ci-espidf into the runner's rootless image store
- config.yaml.j2: docker:// labels (per-runner overridable), docker_host
  -> rootless socket, force_pull false
- act_runner.service.j2: XDG_RUNTIME_DIR + DOCKER_HOST -> user socket
- defaults: uniform capacity:4 (drop the debyl capacity:1 workaround);
  esp_idf_version now tags the espressif/idf-based image
- main.yml: import images.yml, drop the host esp-idf install (firmware jobs
  use the espressif/idf job container instead)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TODO.md

@@ -1,52 +0,0 @@
-# TODO
-
-## ✅ Caddy Migration - COMPLETED
-- [x] Migrate from nginx + ModSecurity to Caddy
-- [x] Automatic HTTPS certificate provisioning
-- [x] All sites working with proper IP restrictions
-- [x] Remove migration_mode logic - Caddy is now default
-
-## Infrastructure Cleanup Tasks
-
-### ✅ Phase 1: System LetsEncrypt to Caddy Migration - COMPLETED
-- [x] ~~Create dedicated Caddy certificates volume~~ - Not needed, Caddy manages in /data
-- [x] ~~Copy existing system LetsEncrypt certificates~~ - Not needed, Caddy generated new ones
-- [x] ~~Set proper permissions~~ - Already correct, Caddy runs as podman user
-- [x] Remove LetsEncrypt cron jobs from Ansible (cleanup.yml created)
-- [x] Remove LetsEncrypt cron jobs from remote host (both weekly + 5min jobs removed)
-- [x] Disable ssl role tasks and certificate generation (disabled in deploy_home.yml)
-- [x] ~~Remove certbot installation from common role~~ - Not installed there
-- [x] Uninstall certbot/letsencrypt packages from remote host (removed via dnf)
-- [x] Stop any running LetsEncrypt services (certbot.timer not running)
-- [x] Backup and remove /etc/letsencrypt directory (backup created, directory removed)
-- [x] Remove /srv/http/letsencrypt directory (webroot removed)
-
-### ✅ Phase 2: nginx + ModSecurity Cleanup - COMPLETED
-- [x] Remove nginx container configuration and tasks (deleted all conf-nginx*.yml, nginx.yml)
-- [x] Remove nginx configuration templates and files (removed entire templates/nginx/ directory)
-- [x] Remove ModSecurity rules and configuration (removed from defaults/main.yml variables)
-- [x] Remove nginx/ModSecurity volume mounts and directories (nginx volume backed up and removed)
-- [x] Clean up nginx-related variables from defaults/main.yml (nginx_path removed)
-- [x] ~~Remove firewall rules for nginx~~ - Not needed, Caddy uses same ports
-- [x] Remove nginx systemd services from remote host (container-nginx service removed)
-- [x] ~~Uninstall nginx/ModSecurity packages~~ - Were never system-installed, container-only
-- [x] Clean up nginx log directories and files (/var/log/nginx, /var/log/modsecurity removed)
-- [x] Remove ModSecurity installation directories (/usr/share/modsecurity, /usr/share/coreruleset removed)
-- [x] Create backup of nginx configuration (nginx-backup-{timestamp}.tar.gz created)
-
-### ✅ Phase 3: Final Cleanup - COMPLETED
-- [x] Remove Drone CI infrastructure and ci.bdebyl.net host
-  - [x] Remove Drone container from podman configuration (drone.yml deleted)
-  - [x] Remove ci.bdebyl.net from Caddyfile (site configuration removed)
-  - [x] Clean up drone-related volumes and data (drone volume backed up and removed)
-  - [x] Update firewall rules to remove CI ports (ports were not explicitly opened)
-- [x] Review and remove unused variables and templates
-  - [x] Removed ci_server_name variable
-  - [x] Removed drone-related variables (drone_path, drone_server_proto, etc.)
-  - [x] Cleaned up nginx handler in handlers/main.yml
-  - [x] Updated firewall.yml comments
-- [x] Update documentation to reflect Caddy as web server
-  - [x] Updated CLAUDE.md container organization section
-  - [x] Updated tagging strategy (nginx→caddy, drone marked decommissioned)
-  - [x] Updated target environment description (nginx→Caddy)
-- [x] Verify all services working after cleanup (sites tested and working)

ansible/roles/gitea-actions/defaults/main.yml

@@ -3,23 +3,35 @@ gitea_runner_user: gitea-runner
 gitea_runner_home: /home/gitea-runner
 gitea_runner_version: "0.2.13"
 gitea_runner_arch: linux-amd64
+
+# Max concurrent jobs per runner. Each job runs in its own ephemeral container
+# (docker:// labels backed by rootless podman), so jobs no longer share the
+# gitea-runner user's Go caches and can run fully in parallel without corruption.
 gitea_runner_capacity: 4
 
-# Multiple Gitea instances to run actions runners for
+# Gitea instances to run actions runners for. Override `labels` or `capacity`
+# per runner here if needed.
 gitea_runners:
   - name: debyl
     instance_url: https://git.debyl.io
   - name: skudak
     instance_url: https://git.skudak.com
 
-# Old single-instance format (replaced by gitea_runners list above):
-# gitea_instance_url: https://git.debyl.io
-
 # Paths
 act_runner_bin: /usr/local/bin/act_runner
 act_runner_config_dir: /etc/act_runner
 act_runner_work_dir: /var/lib/act_runner
 
-# ESP-IDF configuration
+# Job container images (built locally into the gitea-runner rootless image
+# store by tasks/images.yml; never pulled — force_pull is false).
+gitea_ci_image: localhost/gitea-ci:latest
+# ESP-IDF firmware image tag tracks the upstream espressif/idf release we build from.
 esp_idf_version: v5.4.1
-esp_idf_path: /opt/esp-idf
+gitea_ci_espidf_image: "localhost/gitea-ci-espidf:{{ esp_idf_version }}"
+
+# Default labels for every runner — map runs-on values to the local CI image.
+# Firmware jobs opt into the ESP-IDF image per-job via `container:` in their workflow.
+gitea_runner_labels:
+  - "fedora:docker://{{ gitea_ci_image }}"
+  - "ubuntu-latest:docker://{{ gitea_ci_image }}"
+  - "ubuntu-22.04:docker://{{ gitea_ci_image }}"

ansible/roles/gitea-actions/handlers/main.yml

@@ -6,16 +6,3 @@
     state: restarted
     daemon_reload: true
   loop: "{{ gitea_runners }}"
-
-- name: restart podman socket
-  become: true
-  ansible.builtin.systemd:
-    name: podman.socket
-    state: restarted
-    daemon_reload: true
-
-- name: restore esp-idf selinux context
-  become: true
-  ansible.builtin.command:
-    cmd: restorecon -R {{ esp_idf_path }}
-  changed_when: true

ansible/roles/gitea-actions/tasks/deps.yml

@@ -1,38 +1,69 @@
 ---
-- name: install podman-docker for docker CLI compatibility
+- name: install podman for rootless CI job containers
   become: true
   ansible.builtin.dnf:
     name:
-      - podman-docker
-      - golang
+      - podman
     state: present
   tags: gitea-actions
 
-- name: create podman socket override directory
+- name: look up gitea-runner uid
   become: true
-  ansible.builtin.file:
-    path: /etc/systemd/system/podman.socket.d
-    state: directory
-    mode: "0755"
+  changed_when: false
+  check_mode: false
+  ansible.builtin.command: id -u {{ gitea_runner_user }}
+  register: gitea_runner_id
+  tags:
+    - gitea-actions
+    - always
+
+- name: set gitea_runner_uid fact
+  ansible.builtin.set_fact:
+    gitea_runner_uid: "{{ gitea_runner_id.stdout | trim }}"
+  tags:
+    - gitea-actions
+    - always
+
+# Rootless podman needs subuid/subgid ranges for the runner user. Fedora's
+# useradd normally assigns them automatically; ensure they exist regardless.
+- name: check gitea-runner subuid mapping
+  become: true
+  ansible.builtin.command: grep -q "^{{ gitea_runner_user }}:" /etc/subuid
+  register: gitea_runner_subuid
+  changed_when: false
+  failed_when: false
   tags: gitea-actions
 
-- name: configure podman socket for gitea-runner access
+- name: assign subuid/subgid ranges for gitea-runner
   become: true
-  ansible.builtin.copy:
-    dest: /etc/systemd/system/podman.socket.d/override.conf
-    content: |
-      [Socket]
-      SocketMode=0660
-      SocketGroup={{ gitea_runner_user }}
-    mode: "0644"
-  notify: restart podman socket
+  ansible.builtin.command: >-
+    usermod
+    --add-subuids 100000000-100065535
+    --add-subgids 100000000-100065535
+    {{ gitea_runner_user }}
+  when: gitea_runner_subuid.rc != 0
+  register: gitea_runner_subuid_added
   tags: gitea-actions
 
-- name: enable system podman socket
+- name: migrate gitea-runner podman storage to new id mapping
   become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.command: podman system migrate
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+  when: gitea_runner_subuid_added is changed
+  changed_when: true
+  tags: gitea-actions
+
+- name: enable rootless podman socket for gitea-runner
+  become: true
+  become_user: "{{ gitea_runner_user }}"
   ansible.builtin.systemd:
     name: podman.socket
-    daemon_reload: true
+    scope: user
     enabled: true
     state: started
+    daemon_reload: true
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
   tags: gitea-actions

ansible/roles/gitea-actions/tasks/esp-idf.yml

@@ -1,92 +0,0 @@
----
-- name: install ESP-IDF build dependencies
-  become: true
-  ansible.builtin.dnf:
-    name:
-      - git
-      - wget
-      - flex
-      - bison
-      - gperf
-      - python3
-      - python3-pip
-      - cmake
-      - ninja-build
-      - ccache
-      - libffi-devel
-      - libusb1-devel
-    state: present
-  tags: gitea-actions
-
-- name: check if ESP-IDF is installed
-  ansible.builtin.stat:
-    path: "{{ esp_idf_path }}"
-  register: esp_idf_dir
-  tags: gitea-actions
-
-- name: clone ESP-IDF repository
-  become: true
-  ansible.builtin.git:
-    repo: https://github.com/espressif/esp-idf.git
-    dest: "{{ esp_idf_path }}"
-    version: "{{ esp_idf_version }}"
-    recursive: true
-    force: false
-  when: not esp_idf_dir.stat.exists
-  tags: gitea-actions
-
-- name: add ESP-IDF to git safe.directory
-  become: true
-  ansible.builtin.command:
-    cmd: git config --global --add safe.directory {{ esp_idf_path }}
-  changed_when: false
-  tags: gitea-actions
-
-- name: ensure ESP-IDF submodules are initialized
-  become: true
-  ansible.builtin.command:
-    cmd: git submodule update --init --recursive
-    chdir: "{{ esp_idf_path }}"
-  changed_when: false
-  tags: gitea-actions
-
-- name: set ESP-IDF directory ownership
-  become: true
-  ansible.builtin.file:
-    path: "{{ esp_idf_path }}"
-    owner: "{{ gitea_runner_user }}"
-    group: "{{ gitea_runner_user }}"
-    recurse: true
-  tags: gitea-actions
-
-- name: set SELinux context for ESP-IDF directory
-  become: true
-  community.general.sefcontext:
-    target: "{{ esp_idf_path }}(/.*)?"
-    setype: usr_t
-    state: present
-  when: ansible_selinux.status == "enabled"
-  notify: restore esp-idf selinux context
-  tags: gitea-actions
-
-- name: create ESP-IDF tools directory for runner user
-  become: true
-  ansible.builtin.file:
-    path: "{{ gitea_runner_home }}/.espressif"
-    state: directory
-    owner: "{{ gitea_runner_user }}"
-    group: "{{ gitea_runner_user }}"
-    mode: "0755"
-  tags: gitea-actions
-
-- name: install ESP-IDF tools for runner user
-  become: true
-  become_user: "{{ gitea_runner_user }}"
-  ansible.builtin.shell: |
-    export IDF_TOOLS_PATH="{{ gitea_runner_home }}/.espressif"
-    {{ esp_idf_path }}/install.sh esp32
-  args:
-    creates: "{{ gitea_runner_home }}/.espressif/python_env"
-  environment:
-    HOME: "{{ gitea_runner_home }}"
-  tags: gitea-actions

ansible/roles/gitea-actions/tasks/images.yml

@@ -0,0 +1,55 @@
+---
+- name: create CI image build directory
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.file:
+    path: "{{ gitea_runner_home }}/ci-images"
+    state: directory
+    mode: "0755"
+  tags: gitea-actions
+
+- name: stage default CI Containerfile
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.template:
+    src: Containerfile.ci
+    dest: "{{ gitea_runner_home }}/ci-images/Containerfile.ci"
+    mode: "0644"
+  register: ci_containerfile
+  tags: gitea-actions
+
+- name: stage ESP-IDF CI Containerfile
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  ansible.builtin.template:
+    src: Containerfile.espidf.j2
+    dest: "{{ gitea_runner_home }}/ci-images/Containerfile.espidf"
+    mode: "0644"
+  register: espidf_containerfile
+  tags: gitea-actions
+
+- name: build default CI image ({{ gitea_ci_image }})
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  containers.podman.podman_image:
+    name: "{{ gitea_ci_image }}"
+    path: "{{ gitea_runner_home }}/ci-images"
+    build:
+      file: "{{ gitea_runner_home }}/ci-images/Containerfile.ci"
+    force: "{{ ci_containerfile is changed }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+  tags: gitea-actions
+
+- name: build ESP-IDF CI image ({{ gitea_ci_espidf_image }})
+  become: true
+  become_user: "{{ gitea_runner_user }}"
+  containers.podman.podman_image:
+    name: "{{ gitea_ci_espidf_image }}"
+    path: "{{ gitea_runner_home }}/ci-images"
+    build:
+      file: "{{ gitea_runner_home }}/ci-images/Containerfile.espidf"
+    force: "{{ espidf_containerfile is changed }}"
+  environment:
+    XDG_RUNTIME_DIR: "/run/user/{{ gitea_runner_uid }}"
+  tags: gitea-actions

ansible/roles/gitea-actions/tasks/main.yml

@@ -3,7 +3,7 @@
   tags: gitea-actions
 - import_tasks: deps.yml
   tags: gitea-actions
-- import_tasks: esp-idf.yml
+- import_tasks: images.yml
   tags: gitea-actions
 - import_tasks: runner.yml
   tags: gitea-actions

ansible/roles/gitea-actions/tasks/runner.yml

@@ -45,6 +45,8 @@
     mode: "0644"
   vars:
     runner_name: "{{ item.name }}"
+    runner_capacity: "{{ item.capacity | default(gitea_runner_capacity) }}"
+    runner_labels: "{{ item.labels | default(gitea_runner_labels) }}"
   loop: "{{ gitea_runners }}"
   notify: restart act_runner services
   tags: gitea-actions

ansible/roles/gitea-actions/tasks/user.yml

@@ -7,8 +7,6 @@
     shell: /bin/bash
     createhome: true
     home: "{{ gitea_runner_home }}"
-    groups: docker
-    append: true
   tags: gitea-actions
 
 - name: check if gitea-runner lingering enabled

ansible/roles/gitea-actions/templates/Containerfile.ci

@@ -0,0 +1,24 @@
+# Default Gitea Actions job image (managed by ansible: roles/gitea-actions).
+# Covers Go/web/node jobs plus `docker build` (talks to the mounted rootless
+# podman socket). Go toolchains are provided per-job by actions/setup-go.
+FROM node:20-bookworm-slim
+
+ARG DOCKER_CLI_VERSION=27.3.1
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        ca-certificates curl git openssh-client make build-essential \
+        python3 python3-pip jq unzip \
+    && rm -rf /var/lib/apt/lists/*
+
+# Static docker client (no daemon) for jobs that run `docker build` against the
+# mounted podman socket (/var/run/docker.sock).
+RUN curl -fsSL "https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_CLI_VERSION}.tgz" \
+        | tar -xz -C /tmp \
+    && install -m0755 /tmp/docker/docker /usr/local/bin/docker \
+    && rm -rf /tmp/docker
+
+# AWS CLI v2 — several workflows upload artifacts / deploy Lambda.
+RUN curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip \
+    && unzip -q /tmp/awscliv2.zip -d /tmp \
+    && /tmp/aws/install \
+    && rm -rf /tmp/aws /tmp/awscliv2.zip

ansible/roles/gitea-actions/templates/Containerfile.espidf.j2

@@ -0,0 +1,16 @@
+# ESP-IDF firmware job image (managed by ansible: roles/gitea-actions).
+# Adds node (required by actions/checkout and other JS actions) and the AWS CLI
+# (firmware artifacts ship to S3) on top of the official Espressif toolchain.
+# IDF lives at /opt/esp/idf — firmware jobs source /opt/esp/idf/export.sh.
+FROM espressif/idf:{{ esp_idf_version }}
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        curl ca-certificates unzip \
+    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscliv2.zip \
+    && unzip -q /tmp/awscliv2.zip -d /tmp \
+    && /tmp/aws/install \
+    && rm -rf /tmp/aws /tmp/awscliv2.zip

ansible/roles/gitea-actions/templates/act_runner.service.j2

@@ -1,7 +1,7 @@
 [Unit]
 Description=Gitea Actions runner ({{ runner_name }})
 Documentation=https://gitea.com/gitea/act_runner
-After=network.target podman.socket
+After=network.target
 
 [Service]
 ExecStart={{ act_runner_bin }} daemon --config {{ act_runner_config_dir }}/config-{{ runner_name }}.yaml
@@ -10,7 +10,8 @@ TimeoutSec=0
 RestartSec=10
 Restart=always
 User={{ gitea_runner_user }}
-Environment="DOCKER_HOST=unix:///run/podman/podman.sock"
+Environment="XDG_RUNTIME_DIR=/run/user/{{ gitea_runner_uid }}"
+Environment="DOCKER_HOST=unix:///run/user/{{ gitea_runner_uid }}/podman/podman.sock"
 
 [Install]
 WantedBy=multi-user.target

ansible/roles/gitea-actions/templates/config.yaml.j2

@@ -3,27 +3,32 @@ log:
 
 runner:
   file: {{ act_runner_work_dir }}/{{ runner_name }}/.runner
-  capacity: {{ gitea_runner_capacity | default(4) }}
+  capacity: {{ runner_capacity | default(gitea_runner_capacity) | default(4) }}
   timeout: 3h
   insecure: false
   fetch_timeout: 5s
   fetch_interval: 2s
   labels:
-    - ubuntu-latest:host
-    - ubuntu-22.04:host
-    - fedora:host
+{% for label in runner_labels | default(gitea_runner_labels) %}
+    - {{ label }}
+{% endfor %}
 
 cache:
   enabled: true
   dir: {{ act_runner_work_dir }}/{{ runner_name }}/cache
 
 container:
+  # Each job runs in its own ephemeral container (docker:// labels) backed by
+  # the gitea-runner user's rootless podman socket — this is what isolates the
+  # per-job Go module/build caches and fixes cross-repo cache poisoning.
   network: host
   privileged: false
   options:
   workdir_parent:
   valid_volumes: []
-  docker_host: ""
+  # Point act at the real rootless socket so it mounts the correct path into
+  # job containers (the documented rootless-podman gotcha).
+  docker_host: "unix:///run/user/{{ gitea_runner_uid }}/podman/podman.sock"
   force_pull: false
 
 host:

ansible/roles/podman/tasks/main.yml

@@ -78,9 +78,15 @@
 
 - import_tasks: containers/debyltech/fulfillr.yml
   vars:
-    image: git.debyl.io/debyltech/fulfillr:20260603.0222
+    image: git.debyl.io/debyltech/fulfillr:20260605.2021
   tags: debyltech, fulfillr
 
+# Staging back-office (fulfillr-dev.debyltech.com) — same image, staging Turso config.
+- import_tasks: containers/debyltech/fulfillr-dev.yml
+  vars:
+    image: git.debyl.io/debyltech/fulfillr:20260606.0357
+  tags: debyltech, fulfillr-dev
+
 - import_tasks: containers/debyltech/uptime-kuma.yml
   vars:
     image: docker.io/louislam/uptime-kuma:2.3.2