Skip to content

fix: build release binaries from the source tree, not the checkout - #1317

Merged
jeremi merged 1 commit into
mainfrom
fix/release-build-source-checkout-independence
Sep 22, 2026
Merged

jeremi merged 1 commit into
mainfrom
fix/release-build-source-checkout-independence

Conversation

@jeremi

@jeremi jeremi commented Sep 22, 2026

Copy link
Copy Markdown
Member

Pull Request

Summary

Release area. A release binary must be a function of the source tree, not of
the checkout it was built in. Four crates in Cargo.lock run
git rev-parse HEAD in their build script and bake the answer into what they
compile:

crate reads embeds as fallback when git fails
wasmtime-internal-cache 48.0.2 full commit COMPILER_VERSION CARGO_PKG_VERSION
cranelift-codegen 0.135.2 first 9 chars VERSION = 0.135.2-<rev> bare 0.135.2
typst-utils 0.15.1 full commit TYPST_COMMIT_SHA unset
wasm-bindgen-shared 0.2.x first 9 chars WBG_VERSION unset (option_env!)

Each is meant to read its own development checkout. The canonical builder
mounts this repository at /workspace and puts CARGO_HOME inside it, so
repository discovery from a vendored build script found /workspace/.git and
recorded the commit being released. Release bytes therefore changed on every
commit even when no compiled input changed.

This runs the builder container with GIT_CEILING_DIRECTORIES=/workspace,
which stops repository discovery at the mount point.

Why this is blocking v0.33.0

The Base Registry image advisory baseline records reviewed layer digests in
one commit and compares them, via whole_image_fingerprint_equals, against a
candidate built from a later commit. Candidate run 35747918154 (source
125f67b37) failed exactly there:

exposure assertion false: affected=['CVE-2026-19499','CVE-2026-5435','CVE-2026-85091']: ordered OCI rootfs.diff_ids changed
advisory baseline: grype image=sha256:fc6b63fb... threshold=high findings=20 blocking=3 exceptions=3 invalid=3

Layers 0-20 and 22 were byte-identical to the reviewed base; only the BReg
application layer moved, and inside it only /usr/local/bin/breg. The two
occurrences in that binary were located at offsets 1472394
((engine ..)-125f67b37b424..., wasmtime, full 40) and 2195806
(0.135.2-125f67b37, cranelift-codegen, 9 chars). No compiled input changed
between the reviewed revision 6a460c2cf and 125f67b37.

The assertion could never converge again, and no renewal of the baseline
could succeed, because every renewal commit changes the bytes it is
renewing against. This makes the gate convergent again.

Nothing else in the release path sees this. The weekly repeatability proof
rebuilds one published tag, so the commit is identical on both sides of its
comparison and the defect is invisible to it.

Blast radius

GIT_CEILING_DIRECTORIES is a git-internal discovery limit, not a build flag.
All four crates degrade to their packaged-build value rather than failing, so
no build breaks. Verified there is no other git reader:

  • No workspace build.rs reads git. registry-platform-buildinfo/build.rs
    embeds only DISPLAY_VERSION and IS_RELEASE_BUILD, both from environment.
  • No vergen, built, git2, gix, or shadow-rs in Cargo.lock.
  • Of the six candidate images, only BReg contained a commit string in either
    the full-40 or 9-character form (breg: 1 full, 2 abbreviated; casework,
    discovery, evidence, relay, scheduling: zero of both).

typst-utils and wasm-bindgen-shared are in the lock but not in any
release asset; they are listed for completeness.

Verified empirically on git 2.54.0 that a ceiling equal to the repository root
blocks git rev-parse HEAD from a subdirectory (exit 128, "not a git
repository") while still resolving with the cwd at the root itself. Cargo runs
build scripts below /workspace and runs itself at /workspace, so this is
the property required: build scripts are blocked, Cargo's own git access is
not.

Adjacent addition, easy to drop

check_source_commit_absent reads every staged binary back before the payload
is checksummed and fails the build if one still contains the exact source
commit. It is a regression check for a path the ceiling does not cover, for
example a build script that names the repository directory itself rather than
discovering it. It matches the full commit only, so a build script embedding
an abbreviation alone passes here and is caught by the ceiling instead. The
commit is supplied as RELEASE_SOURCE_COMMIT, taken from RELEASE_SOURCE_SHA
on the release path and from the local checkout otherwise; when neither
resolves, the check says so on stderr and passes rather than failing silently.

If this is out of scope for a release-unblocking change, say so and I will
drop the helper, its call site, and its four tests. The ceiling alone fixes
the defect.

Also in the canonical container contract

The inner RELEASE_BUILDER_READY guard now requires
GIT_CEILING_DIRECTORIES=/workspace alongside the existing repo_root,
CARGO_HOME, CARGO_TARGET_DIR, and tag checks. The ceiling is part of the
recipe, not a setting a caller can drop.

Evidence

Written test-first; the four new tests failed with
check_source_commit_absent: command not found before the helper existed.

claim command result
canonical recipe passes the ceiling and the commit to the container, and the inner guard requires the ceiling python3 -m unittest release/scripts/test_zig_glibc_compiler.py OK, 20 tests (4 new, 2 new assertions in the existing dispatch test)
release CLI unaffected python3 -m unittest release/scripts/test_registry_release.py OK, 95 tests
candidate assembly unaffected python3 -m unittest release/scripts/test_release_candidate.py OK
repeatability workflow unaffected python3 -m unittest release/scripts/test_release_repeatability_workflow.py OK
gate inventory unaffected python3 -m unittest release/scripts/test_check_gates_inventory.py OK
release workflow structure unaffected python3 -m unittest release/scripts/test_release_workflow_structure.py OK
rehearsal unaffected python3 -m unittest release/scripts/test_release_rehearsal.py OK
source model unaffected python3 -m unittest release/scripts/test_check_release_source_model.py OK
script parses bash -n release/scripts/build-release-binaries.sh OK
no new lint findings shellcheck release/scripts/build-release-binaries.sh only the pre-existing SC1091 info on sourcing ../glibc-floor.env

shfmt -d is inconclusive: it wants to reformat pre-existing case statements
and [[ ]] continuations throughout the file. Neither shellcheck nor shfmt
runs in this repo's CI and the file's existing style already diverges from
shfmt defaults, so formatting was left alone and the additions follow the
file's 2-space convention.

release_candidate.py only regex-greps this recipe for
cp ... dist/image-bin/<name> lines, so the additions do not affect how the
payload inventory is derived. Both release-candidate.yml and
release-rehearsal.yml already set RELEASE_SOURCE_SHA, so the release path
always knows its exact commit, and both cache keys hash
build-release-binaries.sh, so this change correctly invalidates the build
cache.

The convergence proof is the next candidate itself: after this lands, BReg's
bytes depend only on source content, and the pending advisory-baseline renewal
touches only release/security/*.json and
release/scripts/test_check_advisory_baselines.py, neither of which reaches a
binary.

Notes

Release provenance and security baseline change; needs explicit review.

It changes what goes into published release bytes. COMPILER_VERSION in
wasmtime-internal-cache keys an on-disk module cache, so the fallback makes
that key a function of the wasmtime version rather than of an unrelated
repository's commit, which is the correct behaviour for a shipped binary.
VERSION in cranelift-codegen is diagnostic only. Neither is a security
control.

No published bytes are replaced and no tag moves. The candidate that exposed
this failed before its reverify-and-attest step, so nothing was sealed,
attested, tagged, or published; v0.33.0 will be cut from a new candidate.

Follow-up in this release, not in this PR: renew the BReg advisory baseline
against the post-fix image per release/OPERATIONS.md, then request a fresh
candidate.

DCO

  • Every commit includes a Signed-off-by trailer.
  • I reviewed the submitted changes and am responsible for the contribution.

Four crates in the lock run `git rev-parse HEAD` in their build script and
bake the answer into what they compile. The canonical builder mounts the
repository at /workspace with CARGO_HOME inside it, so they resolved this
repository rather than their own and recorded the commit being released.
Release bytes then changed with every commit even when no compiled input
did, which is what stalled v0.33.0: the Base Registry image advisory
baseline records reviewed layer digests in one commit and compares them
against the candidate built from the next one, so the whole-image
fingerprint assertion could never converge again and no renewal of it
could succeed.

Run the builder container with GIT_CEILING_DIRECTORIES=/workspace so
repository discovery stops at the mount point. All four crates already
fall back to their packaged-build value when git fails, and no workspace
crate reads git, so nothing else changes. The inner invocation now
refuses a payload built without the ceiling, making it part of the
canonical container contract instead of a caller setting.

Also read the staged payload back and fail the build when a binary still
contains the exact source commit, as a regression check for a path the
ceiling does not cover.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@jeremi
jeremi merged commit 82c1b81 into main Sep 22, 2026
43 checks passed
@jeremi
jeremi deleted the fix/release-build-source-checkout-independence branch September 22, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant