feat(skudak-cloud): repair LibreSign, brand its mail, add Redis

LibreSign had been silently broken since it was first deployed in
January. Every step of the old before-starting hook ended in `|| echo`,
so six months of failures logged nothing.

LibreSign repair
- Root cause was a stale config_path: a valid OpenSSL root CA existed at
  generation 1, a failed CFSSL attempt left an empty generation 2, and
  config_path was left pointing at the empty one. Regenerated as
  "Skudak LLP" (was the pre-rename "Skudak Rennsport LLP").
- Deleted the hook. Java/PDFtk/jSignPdf live under data/appdata_*, a
  persisted volume, so they only ever needed installing once. Install and
  verification are now explicit tasks that actually fail.
- PHP_MEMORY_LIMIT 1024M -- the 512M image default fails opaquely
  mid-signature. LC_ALL/LANG so the JVM is not ANSI_X3.4-1968.
- signature_render_mode=GRAPHIC_ONLY. Any other mode halves the stamp
  width and overlays a name/date block that collides with the drawn mark
  and duplicates what our documents already typeset. The value must be
  exactly GRAPHIC_ONLY; a bare "GRAPHIC" is accepted by occ, matches no
  radio in the UI, and silently reverts to default.
- write_qrcode_on_footer=false, written with --type=boolean because
  FooterHandler reads it via getValueBool and the typed appconfig API
  does not coerce a string "0". The validation URL text is kept.
- identification_documents=0 -- the default gates signing behind an ID
  upload plus admin approval, so signers saw no way to sign.
- shareapi_restrict_user_enumeration_full_match=no, so an email owned by
  an existing account can be added as a signer. Root cause is in core
  (MailPlugin.php:163), not LibreSign. Do NOT set full_match_email=no --
  that disables email signer search entirely.

Mail branding (skudakmail app)
- Two supported extension points, no core patch and no LibreSign fork:
  mail_template_class for layout, subjects, button labels and the footer
  LibreSign never adds; and a BeforeMessageSent listener to embed the
  wordmark as a cid: part so it survives remote-image blocking.
- A third listener adds scoped CSS fixing the signing page being clipped
  on iOS Safari (100vh -> 100dvh). Patched upstream too.
- skudakmail-verify.php.j2 asserts all of the above through the real
  useTemplate() path and fails the play on drift. Every assertion was
  proven to fail when deliberately regressed.

Redis
- memcache.locking was unset, so Nextcloud used DBLockingProvider and
  every file lock became a MariaDB write -- the contention behind the
  intermittent multi-second stalls. Verified after: db locks static,
  redis keys growing.
- requirepass lives in a mounted 0640 conf, not --requirepass, which
  would leak it into podman inspect, the systemd unit and ps. The file is
  chowned to uid 999 because redis-server does not run as root and the
  :ro mount stops the image fixing it itself.
- No maxmemory: cache is evictable, locks are NOT, and evicting a held
  lock permits concurrent writers to one file. No persistence either --
  a restored RDB could reinstate locks whose owner is long dead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bastian de Byl
2026-08-01 15:54:52 -04:00
parent c184099b2d
commit fec7d62acb
14 changed files with 1389 additions and 41 deletions
@@ -25,15 +25,17 @@
- name: flush handlers
ansible.builtin.meta: flush_handlers
- name: copy skudak cloud libresign setup script
# The former libresign-setup.sh before-starting hook re-ran
# `occ libresign:install --java/--pdftk/--jsignpdf` on every container start,
# with every line ending in `|| echo`, so six months of failures logged
# nothing. Those binaries live under data/appdata_*/libresign, which IS a
# persisted volume, so they only ever needed installing once. Installation and
# verification are now explicit Ansible tasks below that actually fail.
- name: remove obsolete skudak cloud libresign setup hook
become: true
ansible.builtin.template:
src: nextcloud/libresign-setup.sh.j2
dest: "{{ cloud_skudak_path }}/scripts/libresign-setup.sh"
owner: "{{ podman_subuid.stdout }}"
group: "{{ podman_subuid.stdout }}"
mode: 0755
notify: restorecon podman
ansible.builtin.file:
path: "{{ cloud_skudak_path }}/scripts/libresign-setup.sh"
state: absent
- import_tasks: podman/podman-check.yml
vars:
@@ -63,6 +65,90 @@
vars:
container_name: skudak-cloud-db
# ---------------------------------------------------------------------------
# Redis: Nextcloud distributed cache + transactional file locking.
#
# Without it, memcache.locking is unset and Nextcloud falls back to
# DBLockingProvider (lib/private/Server.php:977) -- every file lock becomes a
# MariaDB write against oc_file_locks. A single directory PROPFIND takes dozens
# of locks and two desktop sync clients issue them continuously, which is the
# contention behind the intermittent multi-second stalls.
#
# Also fixes a second problem: memcache.local is APCu, which is PER-PROCESS,
# and Apache here runs mpm_prefork -- so every child holds its own cold cache.
# A distributed cache is shared across all of them.
#
# MUST be created before skudak-cloud below. The Nextcloud entrypoint writes
# its redis config on start; if the host does not resolve at that moment the
# instance comes up pointing at nothing.
- name: create skudak cloud redis config directory
become: true
ansible.builtin.file:
path: "{{ cloud_skudak_path }}/redis"
state: directory
owner: "{{ podman_subuid.stdout }}"
group: "{{ podman_subuid.stdout }}"
mode: 0755
notify: restorecon podman
- name: template skudak cloud redis config
become: true
ansible.builtin.template:
src: nextcloud/redis-skudak.conf.j2
dest: "{{ cloud_skudak_path }}/redis/redis.conf"
owner: "{{ podman_subuid.stdout }}"
group: "{{ podman_subuid.stdout }}"
mode: 0640
notify: restorecon podman
no_log: true
- name: flush handlers
ansible.builtin.meta: flush_handlers
# The redis:alpine image runs redis-server as uid 999 / gid 1000, NOT root, and
# the config is mounted :ro so the image's own entrypoint cannot chown it --
# it logs "cannot change owner ... Read-only file system" and then dies with
# "Fatal error, can't open config file: Permission denied", crash-looping.
# Nextcloud, already pointed at redis by then, answers HTTP 500.
#
# Same idiom as the `podman unshare chown -R 33:33` for www-data above: map the
# in-container uid through the rootless userns. 0640 owned by 999:1000 keeps
# the password unreadable to other users on the host while letting redis read
# it -- which is the entire reason for using a file over --requirepass.
- name: unshare chown the skudak redis config to the redis uid
become: true
become_user: "{{ podman_user }}"
changed_when: false
ansible.builtin.command: >
podman unshare chown 999:1000 {{ cloud_skudak_path }}/redis/redis.conf
- import_tasks: podman/podman-check.yml
vars:
container_name: skudak-cloud-redis
container_image: "{{ redis_image }}"
- name: create skudak-cloud-redis container
become: true
become_user: "{{ podman_user }}"
containers.podman.podman_container:
name: skudak-cloud-redis
image: "{{ redis_image }}"
restart_policy: on-failure:3
log_driver: journald
network:
- shared
# No `ports:` -- deliberately unpublished. Service discovery is by
# container name over `shared`, the same way MYSQL_HOST reaches
# skudak-cloud-db.
volumes:
- "{{ cloud_skudak_path }}/redis/redis.conf:/etc/redis/redis.conf:ro"
command: redis-server /etc/redis/redis.conf
- name: create systemd startup job for skudak-cloud-redis
include_tasks: podman/systemd-generate.yml
vars:
container_name: skudak-cloud-redis
- import_tasks: podman/podman-check.yml
vars:
container_name: skudak-cloud
@@ -83,11 +169,34 @@
MYSQL_DATABASE: skucloud
MYSQL_HOST: skudak-cloud-db
MYSQL_USER: skucloud
# LibreSign signs PDFs in-process; the image default of 512M is not
# enough and manifests as an opaque failure mid-signature.
PHP_MEMORY_LIMIT: 1024M
PHP_UPLOAD_LIMIT: 512M
# Without these the JVM comes up as ANSI_X3.4-1968 and LibreSign's
# config check warns that accented characters in signer names will be
# mangled. See LibreSign issue #4872.
LC_ALL: C.UTF-8
LANG: C.UTF-8
# These three env vars are the WHOLE redis wiring. The image ships
# config/redis.config.php, which -- when REDIS_HOST is set -- declares
# memcache.distributed, memcache.locking AND the connection block.
# Verified against the copy in this instance's persisted config volume.
#
# Do NOT also `occ config:system:set` those keys. occ writes config.php,
# but Nextcloud merges every *.config.php drop-in AFTER it, so the
# drop-in wins -- config.php would read as authoritative while being
# silently overridden. memcache.local stays APCu (apcu.config.php).
#
# REDIS_HOST_PASSWORD_FILE is NOT usable here: the drop-in in this
# volume predates that feature and reads only REDIS_HOST_PASSWORD.
REDIS_HOST: skudak-cloud-redis
REDIS_HOST_PORT: "6379"
REDIS_HOST_PASSWORD: "{{ cloud_skudak_redis_pass }}"
volumes:
- "{{ cloud_skudak_path }}/apps:/var/www/html/custom_apps"
- "{{ cloud_skudak_path }}/data:/var/www/html/data"
- "{{ cloud_skudak_path }}/config:/var/www/html/config"
- "{{ cloud_skudak_path }}/scripts/libresign-setup.sh:/docker-entrypoint-hooks.d/before-starting/libresign-setup.sh:ro"
ports:
- "8090:80"
@@ -96,20 +205,208 @@
vars:
container_name: skudak-cloud
# Install poppler-utils for pdfsig/pdfinfo (LibreSign handles java/pdftk/jsignpdf via occ)
# This needs to be reinstalled on each container recreation
- name: install poppler-utils in skudak-cloud
# ---------------------------------------------------------------------------
# LibreSign (e-signature for Skudak agreements)
#
# poppler-utils supplies pdfsig/pdfinfo; ghostscript is used for PDF
# normalisation. Both land in /usr, which is NOT a persisted volume, so they
# must be reinstalled after every container recreation. Java, PDFtk and
# jSignPdf are different -- LibreSign installs those under
# data/appdata_*/libresign, which IS persisted, so they survive.
- name: install libresign runtime dependencies in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command:
cmd: >
podman exec -u 0 skudak-cloud
sh -c "apt-get update && apt-get install -y --no-install-recommends
poppler-utils && rm -rf /var/lib/apt/lists/*"
register: poppler_install
changed_when: "'is already the newest version' not in poppler_install.stdout"
poppler-utils ghostscript && rm -rf /var/lib/apt/lists/*"
register: libresign_deps
changed_when: "'is already the newest version' not in libresign_deps.stdout"
# When the container is recreated, the entrypoint re-extracts Nextcloud into
# the /var/www/html volume before Apache starts. Every occ call below races
# that: it fails with "Failed opening required .../lib/versioncheck.php" until
# the extraction completes. Poll until occ answers rather than sleeping a
# fixed interval, which would be both slower and still unreliable.
- name: wait for nextcloud to be ready in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ status --output=json
register: skudak_occ_ready
until: skudak_occ_ready.rc == 0 and 'installed' in skudak_occ_ready.stdout
retries: 30
delay: 5
changed_when: false
# A disabled app deregisters every `occ libresign:*` command, which makes the
# app look uninstalled rather than switched off. Found disabled on 2026-07-31.
- name: ensure libresign app is enabled in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ app:enable libresign
register: libresign_enable
changed_when: "'already enabled' not in libresign_enable.stdout"
# Ensure-installed: LibreSign no-ops when the binaries are already present
# under data/appdata_*/libresign (a persisted volume). It prints "Finished with
# success." either way and gives no signal distinguishing a fresh download from
# a no-op, so this never reports changed rather than reporting it every run.
- name: install libresign java/pdftk/jsignpdf binaries in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ libresign:install --java --pdftk --jsignpdf
register: libresign_install
changed_when: false
failed_when: "'Finished with success' not in libresign_install.stdout"
- name: check whether libresign root certificate is configured
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ libresign:configure:check --certificate
register: libresign_cert_check
changed_when: false
failed_when: false
# Guarded deliberately. Running this unconditionally would mint a new root CA
# on every deploy and invalidate every certificate already issued to a signer,
# breaking the trust chain on documents that were already signed.
#
# Do NOT add --ou here: LibreSign appends its own `libresign-ca-id:...` entry
# to the OU field, and the combined value overruns the 64-character ASN.1
# limit for organizationalUnitName, failing with "string too long".
- name: generate libresign root certificate for skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ libresign:configure:openssl
--cn="{{ libresign_skudak_cert_cn }}"
-o "{{ libresign_skudak_cert_o }}"
-c "{{ libresign_skudak_cert_c }}"
-s "{{ libresign_skudak_cert_st }}"
-l "{{ libresign_skudak_cert_l }}"
when: "'error' in libresign_cert_check.stdout"
changed_when: true
# LibreSign defaults to requiring every signer to upload an identification
# document, which then needs approval by a member of `approval_group` before
# the sign action unlocks. For three partners signing their own partnership
# instruments that is pure friction -- the emailed invitation is the identity
# check. Without this, signers see "Upload file" and no way to sign.
- name: relax libresign identification-document gate in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:app:set libresign identification_documents --value=0
register: libresign_ident
changed_when: "'is now set to' in libresign_ident.stdout"
# SIGNAME_AND_DESCRIPTION (the LibreSign default) typesets the signer's NAME as
# text and offers no drawing surface at all. GRAPHIC is the mode that asks for
# an actual signature graphic -- drawn, uploaded or typed -- and stamps ONLY
# that mark, with no description block.
#
# Deliberately GRAPHIC_ONLY rather than the LibreSign default of
# GRAPHIC_AND_DESCRIPTION. In the latter,
# SignatureTextService::getSignatureWidth() returns `$current / 2` whenever a
# text template is set, splitting the stamp into a graphic half and a text
# half. Our documents already typeset the signer's printed name and the date
# either side of the signature rule (\signatureblock in skudak-contract.cls),
# so LibreSign's own name/date block is both redundant and prone to colliding
# with the drawn mark. GRAPHIC_ONLY takes the early return in that method,
# using the full width to stamp the signature alone.
#
# The value MUST be exactly 'GRAPHIC_ONLY' -- see
# SignerElementsService::RENDER_MODE_GRAPHIC_ONLY (line 25). 'GRAPHIC' is NOT
# a valid constant; setting it writes a value the admin UI cannot match to any
# radio button, silently reverting the effective behaviour to the default.
- name: use signature-only stamp in skudak-cloud libresign
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:app:set libresign signature_render_mode --value=GRAPHIC_ONLY
register: libresign_render
changed_when: "'is now set to' in libresign_render.stdout"
# LibreSign stamps a validation footer onto EVERY page of a signed PDF
# (FooterHandler::getFooter(), default on). The QR block within it is a large
# square that lands in the same band as our own document footer -- the rule,
# "Page N of M" and the Skudak mark set by skudak-contract.cls -- and overlaps
# it.
#
# The QR is dropped; the "Digitally signed by ... Validate in <url>" TEXT is
# deliberately KEPT. That line is how a recipient independently verifies who
# signed, when, and under which certificate, which matters for instruments that
# may have to stand up in diligence. Only the redundant graphic goes -- the URL
# it encodes remains printed beside it.
#
# Must be written with --type=boolean: FooterHandler reads it via
# getValueBool() (line 158), and the typed appconfig API does not coerce a
# string "0" to false.
# Lets an email that belongs to an existing Nextcloud account be added as a
# LibreSign signer. Arbitrary external addresses already worked; ONLY
# account-owned ones failed, with a bare "No signers." and nothing logged.
#
# Root cause is in Nextcloud core, not LibreSign --
# lib/private/Collaboration/Collaborators/MailPlugin.php:128-164. On an exact
# email match against the local system address book, with
# shareeEnumerationFullMatch on (its default), the plugin adds a TYPE_USER
# result and returns false. LibreSign registers that plugin as
# MailByMailPlugin with shareType = TYPE_EMAIL, so the TYPE_USER branch is
# skipped, nothing is added, and the early return still fires -- never
# reaching line 243 where the free-form email result is synthesised.
#
# Safe here: shareapi_allow_share_dialog_user_enumeration is already at its
# default 'yes', so users are discoverable by partial search regardless. This
# changes how exact matches are handled, not who can be found.
#
# DO NOT set shareapi_restrict_user_enumeration_full_match_email to 'no'. That
# hits an early bail at MailPlugin.php:67-69 and disables email signer search
# ENTIRELY, including the arbitrary-address case that works today. The verify
# script asserts it has not been set that way.
- name: allow account-owned emails as libresign signers
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:app:set core
shareapi_restrict_user_enumeration_full_match --value=no
register: skudak_enum_fullmatch
changed_when: "'is now set to' in skudak_enum_fullmatch.stdout"
- name: drop libresign validation QR code from signed-PDF footer
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:app:set libresign write_qrcode_on_footer
--value=0 --type=boolean
register: libresign_qr
changed_when: "'is now set to' in libresign_qr.stdout"
# The whole point of this block. Previously every step ended in `|| echo`, so
# a broken LibreSign deployed clean and stayed broken for six months.
- name: verify libresign configuration in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ libresign:configure:check
register: libresign_verify
changed_when: false
failed_when: libresign_verify.stdout is search('\berror\b')
- name: disable nextcloud signup link in config
become: true
ansible.builtin.lineinfile:
@@ -155,6 +452,111 @@
changed_when: "'System config value log_rotate_size' in skudak_log_rotate.stdout"
failed_when: false
# ---------------------------------------------------------------------------
# Skudak mail branding
#
# custom_apps IS a persisted bind mount, so the app survives container
# recreation; only enabling it and the config values need reasserting.
- name: deploy skudakmail email-template app to skudak-cloud
become: true
ansible.builtin.copy:
src: skudakmail/
dest: "{{ cloud_skudak_path }}/apps/skudakmail/"
owner: "{{ podman_subuid.stdout }}"
group: "{{ podman_subuid.stdout }}"
mode: 0644
directory_mode: 0755
notify: restorecon podman
- name: unshare chown skudakmail app
become: true
become_user: "{{ podman_user }}"
changed_when: false
ansible.builtin.command: >
podman unshare chown -R 33:33 {{ cloud_skudak_path }}/apps/skudakmail
- name: enable skudakmail app in skudak-cloud
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud php occ app:enable skudakmail
register: skudakmail_enable
changed_when: "'already enabled' not in skudakmail_enable.stdout"
# Supported extension point -- Mailer::createEMailTemplate() checks this and
# instantiates the named class if it extends EMailTemplate. Not a core patch.
- name: point nextcloud at the skudak email template
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:system:set mail_template_class
--value={{ "OCA\\Skudakmail\\Mail\\SkudakEMailTemplate" }}
register: skudak_mail_class
changed_when: "'set to' in skudak_mail_class.stdout"
# Email asset URLs and every link LibreSign puts in a signature invitation are
# built from overwrite.cli.url when sending from a background job. It pointed
# at the pre-rename cloud.skudakrennsport.com, so invitations carried the old
# domain and the logo <img> resolved against it.
- name: set skudak-cloud canonical cli url
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud
php occ config:system:set overwrite.cli.url
--value=https://{{ cloud_skudak_server_name_new }}
register: skudak_cli_url
changed_when: "'set to' in skudak_cli_url.stdout"
# Theming that the email template reads. The logo MUST be a wide, tightly
# cropped image: Mailer clamps to MAX_LOGO_SIZE=105 preserving aspect, so a
# SQUARE logo renders as a 105x105 block in a coloured band -- which is
# exactly how an 8334x8334 upload turned the header into a giant blue blob.
- name: set skudak-cloud theming
become: true
become_user: "{{ podman_user }}"
ansible.builtin.command: >
podman exec -u www-data skudak-cloud php occ theming:config {{ item.k }} "{{ item.v }}"
loop:
- {k: name, v: "Skudak"}
- {k: slogan, v: "Aftermarket vintage car parts and restoration"}
- {k: url, v: "https://skudak.com"}
- {k: primary_color, v: "{{ theming_skudak_primary }}"}
- {k: background_color, v: "{{ theming_skudak_primary }}"}
register: skudak_theming
changed_when: "'Updated' in skudak_theming.stdout"
loop_control:
label: "{{ item.k }}"
# Branding rides on OC\Mail\EMailTemplate, which is Nextcloud's PRIVATE
# namespace with no API stability guarantee. A Nextcloud major upgrade disables
# the app (info.xml pins max-version), Mailer falls back to the stock template,
# and mail keeps sending -- unbranded and silent. This turns that silence into
# a failed play. Renders through Message::useTemplate(), the real path, and
# also re-asserts the LibreSign signing settings.
- name: template skudakmail verification script
become: true
ansible.builtin.template:
src: nextcloud/skudakmail-verify.php.j2
dest: "{{ cloud_skudak_path }}/scripts/skudakmail-verify.php"
owner: "{{ podman_subuid.stdout }}"
group: "{{ podman_subuid.stdout }}"
mode: 0644
notify: restorecon podman
- name: verify skudak mail branding is live
become: true
become_user: "{{ podman_user }}"
ansible.builtin.shell: >
set -o pipefail;
podman exec -i -u www-data skudak-cloud php
< {{ cloud_skudak_path }}/scripts/skudakmail-verify.php
args:
executable: /bin/bash
register: skudakmail_verify
changed_when: false
- include_tasks: containers/cloud-cron.yml
vars:
cron_name: skudak-cloud
@@ -183,10 +585,18 @@
# cannot repeat. The personal task's `/skudakcloud/**` exclude is PERMANENT --
# it is what keeps business data out of personal storage, not a stopgap.
#
# Still outstanding: the data itself lives on personal TrueNAS hardware. To
# finish separating, add an S3 stage to cloud-backup.sh.j2 guarded by a
# `backup_s3_*` var so only this instance opts in -- awscli2 is already
# installed on the host -- and then drop the TrueNAS rsync below.
# A direct host-to-iDrive S3 stage was built here and then REMOVED on
# 2026-07-31. It would have written the same data into the same `backup-all`
# bucket that the TrueNAS cloud-sync task above already fills -- duplicate
# storage, two writers to one prefix, for no additional coverage. Offsite to
# business-owned storage was already solved by that cloud-sync task; the
# earlier note in this file proposed adding S3 *and then dropping the rsync*,
# i.e. replacement, and building both was a misreading of it.
#
# If offsite is ever moved onto this host, it must REPLACE the rsync below,
# not run beside it. The open question to settle first is whether the
# TrueNAS -> iDrive leg is independently verifiable; nobody has confirmed that
# task's run history end to end, and keeping this chain means trusting it.
- include_tasks: containers/cloud-backup.yml
vars:
backup_name: skudak-cloud
+1
View File
@@ -70,6 +70,7 @@
- import_tasks: containers/skudak/cloud.yml
vars:
db_image: docker.io/library/mariadb:10.6
redis_image: docker.io/redis:8.2-alpine
image: docker.io/library/nextcloud:34.0.2-apache
tags: skudak, skudak-cloud