Repository navigation
ci(ns8): gate releases on what actually reached the registry - #149
Conversation
Two silent-failure guards, both for bugs that already shipped once. Post-publish multi-arch assertion. A release once published an amd64-only image as :latest. It installs on amd64 and hard-fails on arm64, the publish step exited 0, and nothing looked at the registry. packaging/ns8/assert- multiarc.py now runs after push-manifest.sh against the published tags and fails the release unless every required architecture is present, that architecture's config really declares it, it carries an org.nethserver.images label, and the app image that label names is itself multi-arch and contains the same architecture. It checks the registry rather than local storage on purpose: a broken publish must not be able to pass by looking healthy from the runner. It needs no extra secrets, since the module repository is public and the token is anonymous, and it retries briefly so push-to-read propagation latency cannot fail a good release. It runs for branch pushes too, because :edge and :dev are published tags and deserve the same guarantee as a version tag. state-include.conf assertion. That file is handed to restic as --files-from, so restic backs up ONLY the paths listed in it. Dropping a line loses data while the backup job still reports success, which is exactly how #142 shipped the module with the database but not the encryption key. Both paths are now asserted in the validate job, with the reason in a comment, so the next person who trims that file finds out from CI rather than from a failed restore. Also py_compiles the new script in the validate job, since the post-publish assertion cannot run on a PR (there is nothing published to assert against) and would otherwise ship completely unexercised. .gitignore now covers __pycache__, which a local py_compile run would otherwise commit. Verified locally: the assertion script exits 1 on the current amd64-only :latest with an accurate diagnosis, and exits 0 on :edge. The state-include check passes against the real file and detects a simulated dropped line.
There was a problem hiding this comment.
Important
Two things to fix before this can be merged: the assert step as wired fails on every non-PR run, and the app-image check puts the module release behind a workflow it has no dependency on. Both findings below were reproduced against the live GHCR API, not reasoned about.
Reviewed changes
packaging/ns8/assert-multiarc.py(new) — a post-publish gate that reads the GHCR registry (anonymous pull token) instead of local storage and, per tag and per required arch, asserts the tag is an image index, holdslinux/<arch>, that the arch's config really declares it, that the config carriesorg.nethserver.images, and that the app image that label names is itself multi-arch with the same arch. Fetches retry 4 × 10s before failing..github/workflows/ns8-module.yml— runs the assert afterpush-manifest.sh(skipped on PRs, matching the other publish steps),py_compiles it invalidate, and adds astate-include.confguard asserting bothvolumes/calnode-dataandstate/calnode.envare still listed, with the restic--files-fromrationale inline..gitignore— ignores__pycache__/, which a localpy_compilewould otherwise leave committable.
The design reads well and the rationale is right in both places: the registry is the only honest source of truth after a publish, and state-include.conf really is a silent-data-loss surface. The hermetic #148 selftest and this gate are complementary, as the description argues. The issues are in the wiring, not the idea.
ℹ️ A failing gate here reports, it cannot repair
Worth being explicit about the recovery story before this lands: the tags are pushed by push-manifest.sh before the assert step runs, so when the gate goes red the bad or unverified tag is already live and :latest / :edge may already have moved. Re-running only the assert step cannot fix it — recovery means re-pushing the module tags. Not asking for a change here, but the runbook for "NS8 publish went red" should say that.
space-bunny-free (free) | 𝕏
| if: github.event_name != 'pull_request' | ||
| working-directory: packaging/ns8 | ||
| env: | ||
| MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode |
There was a problem hiding this comment.
${{ github.repository_owner }} expands to Calnode, and GHCR's token endpoint rejects a mixed-case scope — so token() in assert-multiarc.py raises HTTPError before any manifest is fetched. As written, every non-PR run of this workflow (every main/dev push, every v*.*.* tag) ends red with cannot fetch …: HTTP 400 — after the tags have already been pushed, and after 30s of retries that can never succeed. Both existing consumers of REPOBASE already normalise for exactly this reason.
| MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode | |
| MODULE_REPO: ghcr.io/${{ github.repository_owner }}/ns8-calnode | |
| IMAGETAGS: ${{ steps.tags.outputs.module_tags }} | |
| run: python3 assert-multiarc.py "${MODULE_REPO,,}" $IMAGETAGS |
| print(f" ok linux/{arch} org.nethserver.images = {app_ref}") | ||
|
|
||
| app_repo, app_tag = split_ref(app_ref) | ||
| abody, actype = fetch_with_retry(app_repo, app_tag, f"app image {app_ref}") |
There was a problem hiding this comment.
This assertion makes a module release depend on docker-publish.yml having finished — both workflows fire on the same push/v*.*.* trigger with no ordering between them, and this fetch gives up after 30s. Because the app image is refetched per arch, a release tag (3 module tags × 2 arches) can burn ~3 minutes and then fail red on a module image that is itself perfectly good, purely because the arm64 app build is still running. Worth keeping the check, but decoupled from the retry budget that protects against push-to-read latency on the module's own tag.

Two silent-failure guards, each for a bug that has already shipped once. Both are the kind that cannot be caught by looking at the code, only by looking at what actually happened.
1. Post-publish multi-arch assertion
A release once published an amd64-only image as
:latest. The publish step exited 0. Nothing looked at the registry. The result installs fine on amd64 and hard-fails on arm64.packaging/ns8/assert-multiarc.pyruns afterpush-manifest.shand fails the release unless, for every required architecture:org.nethserver.images(NS8 reads it at install time; without it the module installs and does nothing)It reads the registry, not local storage, deliberately: a broken publish must not be able to pass by looking healthy from the runner. No extra secrets are needed, since the module repository is public and the token is anonymous. It retries briefly so push-to-read propagation latency cannot fail a good release, and then fails rather than waiting longer.
It runs for branch pushes too.
:edgeand:devare published tags and deserve the same guarantee as a version tag.2.
state-include.confassertionYou asked whether this deserved the cheap guard too. It does, and the case is strong: that file is handed to restic as
--files-from, so restic backs up only the paths listed in it. Dropping a line loses data while the backup job still reports success. That is exactly how #142 shipped a module with the database but not the encryption key.Both paths are now asserted in the validate job, with the reason in the comment so the next person to trim that file learns why they cannot.
On whether the two multi-arch checks are redundant
They are not, they test different things:
:build-<arch>tags and pushing to anoci:path. It publishes nothing.Together they cover the assembly and the result. The missing third piece was exactly this one, which is why an amd64-only
:latestcould ship with both of the others green.Verified locally
The assertion script exits 1 against the current amd64-only
:latestwith an accurate diagnosis, and exits 0 against:edge:The
state-include.confcheck passes against the real file and correctly detects a simulated dropped line. YAML parses, and the build-job step order is login, build, assemble, assert.Notes
validatepy_compiles the script, which otherwise would ship entirely unexercised..gitignorenow covers__pycache__, which a localpy_compilerun would otherwise commit. I hit exactly that while building this.git reset --hardlosing my workflow edits and had to redo them, so the wiring is worth a second pair of eyes.