fix: reject reuse of a failed release publication run at closeout - #1319
Merged
Merged
Conversation
Publishing an already-public release picked the latest workflow_dispatch release.yml run correlated to a healthy docs run and reused it without checking its own conclusion. A publication run whose overall conclusion was failure (for example because its identifier catalog job failed downstream of a successful docs deployment) was silently treated as reusable, so `registry-release publish --plan ... --wait` reported status complete and exit 0 without redispatching anything. Raise instead, naming the run and the exact recovery command (`gh run rerun <run> --failed`), when the correlated publication run completed with a non-success conclusion. Update the OPERATIONS.md failure-handling row, which claimed the closeout path republishes the tag, source commit, and catalog digest; it never redispatches release.yml, so the accurate recovery is to rerun the failed run directly. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
release/OPERATIONS.md's failure-handling table said that when identifierdispatch, deployment, or live smoke fails after publication, rerunning
registry-release publish --plan <candidate-plan.json> --waitmakes thealready-public release "take the exact closeout path and republish the
same tag, source commit, and catalog digest." That is not what the tool
does, and the doc's claim is false for a run whose identifier-catalog job
failed.
Evidence
For v0.33.0,
release.ymlrun 35766749442 succeeded overall except thepublish-identifiersjob ("Publish exact identifier catalog"). Rerunningregistry-release publish --plan <candidate-plan.json> --waitdid notdispatch a new
release.ymlrun and did not redeploy identifiers: itreattached to run 35766749442, waited for the (successful) docs run, ran
verify-public, and printedstatus: completewith exit 0. The only thingthat actually recovered the identifier catalog was
gh run rerun 35766749442 --failed.Root cause
In
release/scripts/registry-release,publish_candidate_plan'salready-published branch looks for an active (in-progress) publication run
first. When none is active, it calls
reusable_docs_publication_runto findthe latest
release.ymlrun correlated to a healthydocs-pages.ymlrun,and reuses that pairing without ever checking the publication run's own
status/conclusion. Sinceverify_public_release.verify()only checks theGitHub Release, its tag, and the published images, and the identifier catalog
is published by a separate
publish-identifiersjob on that samerelease.ymlrun, a
release.ymlrun that failed overall (because that job failed) butwhose docs job still succeeded was accepted as "reusable" and the whole
operation was reported complete.
Fix
reusable_docs_publication_runnow raisesReleasePlanErrornaming the runand the exact recovery command when the correlated publication run completed
with a non-success conclusion, instead of silently treating it as reusable.
This mirrors the same function's existing invariant checks (e.g. the
"resolved to multiple workflow runs" raise) and does not change behavior for
the already-tested case where no docs run exists at all for a failed
publication run.
release/OPERATIONS.md's failure-handling row is correctedto describe what the tool does: recover the bound
release.ymlrun directlywith
gh run rerun <run> --failed, then rerunpublish --waitto verify.No new gates or workflows were added; this only makes an existing check
honest about a run it was already inspecting.
Verification
Added two unit tests to
release/scripts/test_registry_release.py, writtenfirst to confirm they fail against the old code:
test_docs_recovery_rejects_reuse_of_failed_publication_run: callsreusable_docs_publication_rundirectly with a failed publication runpaired with a healthy docs run, and expects
ReleasePlanErrornaming therun ID and
gh run rerun 88 --failed.test_publish_reports_failed_publication_run_instead_of_false_completion:drives
publish_candidate_planend to end for an already-published releasein this exact shape, and expects exit 1 with the recovery command on
stderr, and no docs or publication redispatch.
Ran 97 tests, all passing, clean output.
Test plan
python3 -m unittest release/scripts/test_registry_release.py(97 passed)