Repository navigation
packaging(ns8): read manifest arches from the registry, not local storage - #148
Conversation
…rage
This is the failure that stopped the v0.11.0 module release.
buildah manifest add was given a plain local reference, so it read each
architecture out of local containers-storage. On the release runner buildah
had two graph roots merged into a single store, and the local image record
existed while the manifest blob it pointed at did not:
Error: initializing source containers-storage:[overlay@/home/runner/.local/
share/containers/storage+/run/user/1001/containers]...: reading manifest for
instance "sha256:43c2abfb...": ... file does not exist
The old pre-check missed it. It asked `buildah images -q`, which inspects the
same local store, so it answered "present" for an image whose manifest could
not be read, and the script failed later and far more cryptically. That guard
is gone: a failed add is now the check, and it is authoritative.
The fix is to read each arch from the registry, where build-images.sh has
already pushed it. This is the documented form: the example in
buildah-manifest-add(1) is `buildah manifest add mylist:v1.11 docker://fedora`.
It also means the existing refusal to publish a partial list now inspects the
content that will actually ship.
SOURCE_PREFIX defaults to docker:// and is only overridden by the hermetic
self-test. It uses ${x-default} rather than ${x:-default} on purpose: an empty
value is a deliberate request to read locally, and ${x:-} would silently
replace it.
The CI self-test previously covered only the local-read path, which is exactly
why this shipped broken. It now runs twice: the existing hermetic test, and a
new registry-backed test that assembles from the real published :build-<arch>
tags and pushes to a local oci: path, so it needs no credentials, publishes
nothing, and asserts the assembled index.json really holds two arches.
There was a problem hiding this comment.
Caution
The registry fix itself checks out — I reproduced the new step locally against the real published :build-* tags and the assembly succeeds on buildah 1.33.7. But both new CI paths fail on every run as written: the validate job's subcommand scan now extracts buildah ran from a new comment and exits 1 before any selftest starts, and the registry selftest counts arches in a file that structurally cannot hold them. Two small fixes, both inline.
Reviewed changes
push-manifest.shreads each per-arch image from the registry (docker://<repo>:build-<arch>) instead of local containers-storage, so thebuildah manifest addfailure that broke thev0.11.0release cannot recur; the unreliablebuildah images -qpre-check is gone and a failed add is now the check.- Adds
SOURCE_PREFIX(defaultdocker://, empty string = local), deliberately using${x-default}so the hermetic self-test can ask for a local read. .github/workflows/ns8-module.yml: the existing selftest is renamed "(hermetic)" and passesSOURCE_PREFIX=""explicitly; a second selftest assembles from the publishedghcr.io/calnode/ns8-calnode:build-{amd64,arm64}tags and pushes to a localoci:path, so it needs no credentials and publishes nothing.
For the record, the premise holds: ghcr.io/calnode/ns8-calnode:latest currently serves a single application/vnd.oci.image.manifest.v1+json (amd64 only), while :build-amd64 / :build-arm64 both exist and are anonymously pullable, and the per-arch layers are ~2.2 MiB, so pulling them on every PR costs seconds, not minutes. The README.md local-build recipe already does PUSH=1 on both arches, so it stays correct under the registry read.
ℹ️ Reading :build-<arch> trades local isolation for a shared, mutable tag
The old local read was immune to other runs: each runner's containers-storage held only its own freshly built arches. Reading from the registry means the assembled list is now assembled from whatever :build-amd64 / :build-arm64 point at at that moment, and those tags are rewritten by every branch and release run — concurrency only groups by ref, so a main run and a v* run can overlap and interleave. Narrow, and cancel-in-progress: true closes most of the window, but if you want the guarantee closed, per-run unique build tags (build-<sha>-<arch>) with the public list still assembled from the digests this run pushed would do it. Flagging as a design call, not a blocker.
space-bunny-free (free) | 𝕏
The validate job greps the build scripts for 'buildah [a-z]+' and asserts each extracted word is a real subcommand. My explanatory comment said 'buildah ran with two graph roots...', so the scanner read 'ran' as a subcommand and failed the build. Reworded to 'buildah(1) there had...'. The scanner also matches inside comments generally, so any future sentence phrasing a buildah command in prose can trip it. Not changing that here: it is pre-existing behaviour, and it is exactly the check that caught this.
There was a problem hiding this comment.
Important
One of the two blockers is fixed; the other is still open and the validate job is red on it.
Reviewed changes
- Reworded the
push-manifest.shcomment that thevalidatejob's subcommand scan was tripping on, frombuildah ran with two graph roots…tobuildah(1) there had two graph roots…. I re-ran the scan's own snippet against this head: noMISSINGentries, sobuildah "$sub" --helpno longer fails before the selftests. - Nothing else changed — the registry read, the
SOURCE_PREFIX${x-default}handling, and the two selftests are byte-identical to11e5320.
That single word was the whole of the delta, and it clears the first finding. The second one is untouched: ns8-module.yml:110-113 still counts arches in ${tmp}/out/index.json, which holds a single descriptor for the list and no platform fields, so count is 0 and the step exits 1. Confirmed both ways on this head — a local reproduction of the step against the published :build-* tags gives COUNT=0 against index.json and 2 against the blob it names, and run 38825865179 fails with assembled index holds 0 arches, expected 2.
Technical details
# The registry selftest is still red — same root cause, still open
## Affected sites
- `.github/workflows/ns8-module.yml:110-113` — `jq` counts `.manifests[]` entries carrying a
`platform` in `${tmp}/out/index.json`, which can never hold per-arch entries for a list
## Evidence at head 691682f
- local repro of the step: script exits 0, then `count=0`; the blob named by
`.manifests[0].digest` in `blobs/sha256/<hex>` holds 2
- CI run 38825865179 (`validate`): `##[error]assembled index holds 0 arches, expected 2`
## Why it is not buildah-version specific
`vendor/go.podman.io/image/v5/oci/layout/oci_dest.go` `PutManifest` writes the blob, then
returns early when `instanceDigest != nil`, appending to the index only for the primary
manifest. Instances pushed under `--all` are blobs only.
## Required outcome
The assertion must read the blob referenced by `index.json`, so a wrong or partial assembly
is actually caught — a plain exit-code check would not have caught it, which was the point
of the assertion.
## Suggested approach
```bash
list=$(jq -r '.manifests[0].digest' "${tmp}/out/index.json")
count=$(jq '[.manifests[]
| select(.platform.architecture != null
and .platform.architecture != "unknown")]
| length' "${tmp}/out/blobs/sha256/${list#sha256:}")
```
## Verified
- hermetic path -> 2
- registry path against `ghcr.io/calnode/ns8-calnode:build-{amd64,arm64}` -> 2space-bunny-free (free) | 𝕏
The registry-backed selftest failed on its own assertion, not on the assembly: the run logged 'Published ... (amd64 arm64)' and wrote a correct two-arch list. buildah stores that list as a single entry in the OCI layout's index.json, and the per-arch manifests live inside the index that entry references, so the top level holds no platform entries to count. Follow the reference when the top level is empty, and fail loudly if there is neither a top-level platform set nor a readable blob. Validated locally against a real OCI layout built from the published :edge index.
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
- Reworked the registry selftest's arch assertion in
.github/workflows/ns8-module.yml:113-129so it follows the digest whenindex.jsonitself carries no platform entries: it counts.manifests[]with aplatform, and only on a count of0resolves.manifests[0].digestand recounts inblobs/sha256/<hex>, erroring with the raw index if that blob is missing. - Added
[ .manifests[]? ]to bothjqselectors, so a layout without amanifestsarray reports0and takes the fallback rather than failing on a null iteration.
That is the whole delta since 691682f; the registry read, the SOURCE_PREFIX ${x-default} handling and the two selftests are byte-identical to the previous head. This closes the last open finding from the prior review, so the PR has no remaining threads.
Verified at 0ccb850 on buildah 1.33.7: the registry selftest ran verbatim against the published :build-{amd64,arm64} tags (Published … selftest-registry (amd64 arm64)), index.json came out as the single application/vnd.oci.image.index.v1+json descriptor with top=0, the fallback resolved the blob and returned final count=2. The hermetic selftest still passes, the validate subcommand scan reports no MISSING entries, and the validate and build checks are green on this head.
One thing carried forward rather than raised again: reading :build-<arch> from the registry means the assembled list is built from whatever those tags point at when the step runs, and they are rewritten by every branch and release run while concurrency only groups by ref. Per-run unique build tags would close it. Not a blocker and already flagged on the initial review.
space-bunny-free (free) | 𝕏

Fixes the failure that stopped the
v0.11.0module release.:latestis currently an amd64-only manifest, which works on amd64 and hard-fails on arm64 nodes.The failure
buildah manifest addwas handed a plain local reference, so it read each architecture out of local containers-storage. On the release runner buildah had two graph roots merged into one store, and the local image record existed while the manifest blob it pointed at did not:Why the existing guard missed it
It asked
buildah images -q, which inspects the same local store. So it answered "present" for an image whose manifest could not be read, and the script failed later and far more cryptically. That guard is removed: a failed add is now the check, and it is authoritative. It would also have produced the opposite false answer, reporting a published image as absent, since it never looks at the registry at all.The fix
Read each arch from the registry, where
build-images.shhas already pushed it. This is the documented form, not a workaround. The example inbuildah-manifest-add(1)is:Side benefit: the existing refusal to publish a partial list now inspects the content that actually ships, rather than whatever happens to be in local storage.
SOURCE_PREFIXdefaults todocker://and is overridden only by the hermetic self-test. It uses${x-default}rather than${x:-default}deliberately: an empty value is a real request to read locally, and${x:-}would silently replace it.CI now covers the path that broke
The old self-test only exercised the local-read path, which is precisely why this shipped. It now runs twice:
SOURCE_PREFIX="", pushes tooci:.:build-<arch>tags and pushes to a localoci:path. No credentials, publishes nothing. Asserts the assembledindex.jsonreally holds two arches rather than trusting the exit code.Verified locally
bash -nclean, YAML parses, both selftest steps present,${SOURCE_PREFIX-docker://}confirmed to yielddocker://when unset and empty when explicitly empty, and the buildah subcommand set the CI validator extracts is unchanged. The two-arch registry assembly itself needs a real buildah, so CI is where that gets exercised for real.Not in this PR
The post-publish arch assertion and the README
:latest/arm64 correction were noted as already being handled elsewhere, so I left both alone to avoid colliding. This PR is only the assembly fix and the self-test changes it requires.