Skip to content

security(audit): make the fragment-based audit actually reportable - #446

Open
dormouse-bot wants to merge 6 commits into
mainfrom
fix/audit-fragment-durability
Open

security(audit): make the fragment-based audit actually reportable#446
dormouse-bot wants to merge 6 commits into
mainfrom
fix/audit-fragment-durability

Conversation

@dormouse-bot

@dormouse-bot dormouse-bot commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

#445 gave the audit three subagents that each write a report fragment, but the workflow around them still treats audit-report.md as the only artifact — so the fragments are budgeted for, archived, redacted, and checked by nothing. This wires them in, and raises the job timeout above the orchestrator's own wait deadline so that deadline can actually fire. The review on this PR found one place where that wasn't true — SECURITY.md still said claude_args denies Task/Agent, which the fan-out in #445 made false and which the audit reads as its spec — so that sentence is corrected here, and orchestrator.md gains the reciprocal half of the timeout note. Everything else in both documents already described this behavior; the code now matches.

These are the mechanically-checkable findings from the review on #445 that were still open at merge. Each was re-verified against main at 4b5272b before being acted on. Nothing here is a new opinion about the design.

The four

The 25-minute deadline could never fire. DEADLINE=$(( $(date +%s) + 1500 )) is counted from when the agent starts — after checkout, setup-node, and pnpm install --frozen-lockfile — inside a job declaring timeout-minutes: 20. The runner cancels first, every time, so the graceful "give up and report what the domains found" path is unreachable and an overrun lands as INCONCLUSIVE. That is the exact failure mode the deadline was added to prevent, just reached by the clock instead of by ending the turn. Raised to 40, with the relationship written down so a future edit to either number doesn't silently re-break it.

The fragments were never uploaded. Archive audit transcript had a single path: — the transcript. SECURITY.md line 239 says they ride along with it "so an orchestrator that dies mid-merge still ships whatever the domains found". They now do, alongside audit-report.md. The existing if-no-files-found: warn is what makes this safe: a domain that genuinely died leaves no fragment, and that must not fail the upload.

The redactor didn't cover them. It received only "$TRANSCRIPT" audit-report.md, and the fail-closed rm -f deleted only those two. This is a live violation of the FAIL IF on line 273, which names "audit-report.md, the three per-domain fragments, and the transcript" explicitly. It wasn't a leak only because the fragments weren't published either — which is why it's one fix with the item above, not two.

Nothing enforced the missing-fragment rule. Surface result, file or close issue read audit-status.txt, mapped it to PASS/FAIL/MISSING, and never looked at whether the fragments existed — so a PASS written over a dead domain was published as a pass and opened the release gate in release.yml. That made the FAIL IF on line 274 circular: the only thing enforcing it was the prompt being audited. orchestrator.md line 75 already asserts the reporting step "checks this independently"; it now does.

One judgement call

The guard downgrades such a PASS to INCONCLUSIVE, not FAIL, which is narrower than line 274's "must fail the audit" reads in isolation. The reason is the three-outcome model already in that step: a domain that left no fragment is an audit that didn't finish, not a repo that's insecure, and that step's own comment explains why collapsing those two files an identical issue for "the repo is insecure" and "the auditor stopped early". INCONCLUSIVE already exits non-zero, already files under security-audit-failure, and already holds the release gate shut — so the operational requirement is met either way, and the issue title stays honest about which happened. The INCONCLUSIVE body now names the domains that left no fragment and says the other two still stand. Easy to change to FAIL if you'd rather the document be read literally.

Review follow-up

The review on this PR is applied in ed9d384, which is where the SECURITY.md and orchestrator.md edits above come from. Four changes:

  • The INCONCLUSIVE body is plural-honest. MISSING_FRAGMENTS accumulates a comma-joined list, so "only the named one is unaudited" was wrong for two- and three-domain cases, and the sentence "whatever the other domains reported is below and still stands" was wrong on the path that reaches this branch with nothing below it: the fail-closed rm -f in Redact secrets from agent output deletes all three fragments and audit-report.md while leaving audit-status.txt alone, so a PASS arrives with all three named, no report body, and no transcript link (the archive step matched zero files, so ART_ID is empty). That path is a cleartext-secret incident, so the body now names the redactor when the whole set is missing rather than leaving the reader to infer it from an empty issue.
  • SECURITY.md no longer asserts the opposite of the shipped config. The paragraph beginning "The audit runs as a single headless turn" said claude_args denies Task/Agent; it allows them on purpose and denies only Workflow. That is a live source of a spurious FAIL, since the audit reads this document as its spec, and it contradicted the paragraph two above it that describes the fan-out as intended. Rewritten to say what the config does and to name the actual control — the prompt's non-turn-ending wait, not the allowlist, which only auto-approves.
  • The timeout invariant is written on both sides. DEADLINE=$(( $(date +%s) + 1500 )) in .github/audit/orchestrator.md now carries the reciprocal note, so raising it re-breaks the invariant visibly instead of silently.
  • One fragment list instead of four. A job-level AUDIT_FRAGMENTS feeds the redactor's argv, its fail-closed rm -f, and the guard loop. The artifact path: still spells them out — it wants one entry per line — and carries a pointer back to the env var.

Verification

gh isn't reachable for the reporting step's issue calls, so I exercised the two behavioral changes against fixtures instead of end-to-end — the guard was extracted from the shipped YAML rather than retyped, so the tested text is the text that runs.

Fixture runs

The redactor, with fake secret values planted in the transcript and two fragments:

::warning::Redacted 3 literal secret occurrence(s) from agent output.
step exit: 0
audit-supply-chain.md:           supply fragment leaked *** here
audit-ci-secrets.md:             ci fragment leaked *** here
audit-application.md:            appsec fragment clean
audit-report.md:                 report body, clean
claude-execution-output.json:    {"leak":"***"}

Fail-closed, forcing the redactor to throw partway through: exit 1, and the transcript, report, and both written fragments are all deleted rather than left on disk for the archive step.

The guard, across the cases that matter:

audit-status.txt fragments present result
PASS all three PASS
PASS two MISSING, names audit-application.md
PASS one MISSING, names the other two
PASS none MISSING, names all three
FAIL one FAIL (untouched)
empty all three MISSING, no fragment attribution

The last row is the pre-existing no-status-file path, and it still prints the "exited without writing audit-status.txt" prose rather than the new one — the two causes reach MISSING for different reasons and want different first moves from the reader.

An early version of this diff put the shell comment inside the single-quoted node -e script, where Node parsed it as JavaScript and the step failed closed on every run. bash -n passed it (it's all inside a quoted string); the fixture run is what caught it. Worth knowing that a syntax check alone doesn't cover this step.

Not fixed here

Three findings from that review are application-level rather than audit-plumbing, so they aren't in this PR and remain open — the unbounded #clients / #pairings growth behind the ceremony's ticket cap, accountId normalization being able to mint a never-authorizing ACL record or collapse a lookalike id into the real one, and the verify Funnel probe in deploy/local/install-macos.sh printing a green line when the probe didn't run. Happy to take any of them as its own PR.

Second review follow-up

The second review found that the first bullet above was only half true, and the half it got wrong is the premise of the timeout change: the 25-minute deadline could not fire at 40 minutes either. DEADLINE=$(( $(date +%s) + 1500 )) was recomputed by every shell that ran the wait loop, and 1500s exceeds the ten-minute Bash cap, so the DEADLINE branch was unreachable within any one call and pushed out another 25 minutes by each re-issue §2 itself asks for. Raising timeout-minutes alone bought a longer stall at roughly double the spend, not the graceful give-up it was described as.

6797b29 persists the deadline to ${RUNNER_TEMP:-.}/audit-deadline — written once, read back by every re-issue — so the 25 minutes accumulate and the branch fires on the third call, ~15 minutes inside the 40-minute cancellation. The §2 prose now says to re-issue the block verbatim including those lines, SECURITY.md's FAIL IF names the persistence as part of "a bounded deadline" (the recomputed form satisfied that phrase in words while leaving the runner's cancellation as the only thing ending the wait), and .gitignore covers audit-deadline for the non-CI fallback path. Fixture-verified both ways: a second call reads back the first call's deadline rather than restarting it, and a loop started past its deadline breaks immediately with DEADLINE.

#445 gave the audit three subagents that each write a report fragment,
but the workflow around them still treats `audit-report.md` as the only
artifact. Four consequences, all of which SECURITY.md already claims do
not hold:

- `timeout-minutes: 20` sat below the orchestrator's own 25-minute wait
  deadline, which is counted from when the agent starts. The runner
  cancelled the job first, so the graceful "give up and report what the
  domains found" path was unreachable and every overrun landed as
  INCONCLUSIVE — the failure mode the deadline exists to prevent.
- The fragments were not uploaded, so an orchestrator that died mid-merge
  shipped nothing. SECURITY.md line 239 says they ride along with the
  transcript.
- The redactor covered only the transcript and the merged report, so the
  fragments would have been published unredacted once uploaded. That is a
  live violation of the `FAIL IF` on line 273, which names them.
- Nothing enforced the `FAIL IF` on line 274: the reporting step never
  checked fragment existence, so a `PASS` written over a missing fragment
  was published as a pass and opened the release gate. Only the prompt
  being audited stood between that and a release.

The reporting step now downgrades such a PASS to INCONCLUSIVE rather than
FAIL, matching the three-outcome model already in that step — this is an
audit that did not finish, not a security finding — and names the domains
that left no fragment so the issue says which one is unaudited.

No SECURITY.md or orchestrator.md changes: both already described this
behavior, including orchestrator.md's claim that the reporting step
checks fragments independently. The code now matches the documents.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One finding in the diff, two observations outside it.

The MISSING_FRAGMENTS issue body is written for exactly one missing domain, and two of its sentences are false on paths this PR makes reachable. MISSING_FRAGMENTS accumulates a comma-joined list — the fixture table in the description covers the two-missing and three-missing cases — while the prose says "only the named one is unaudited". The sharper case is the fail-closed path: the rm -f in Redact secrets from agent output deletes all three fragments and audit-report.md but leaves audit-status.txt alone, so a PASS reaches this branch with all three named. The issue then says "Whatever the other domains reported is below and still stands" with nothing below, no transcript link (the archive step matched zero files, so if-no-files-found: warn skipped the artifact and ART_ID is empty), and no mention that the redactor threw — which is the one path where the right first move is to rotate the secrets. Inline suggestion below.

Outside the diff:

  • SECURITY.md still asserts, in the paragraph beginning "The audit runs as a single headless turn", that "claude_args therefore denies the delegation tools (Task/Agent, Workflow)". The shipped claude_args allows Task,Agent and denies only Workflow, and the paragraph beginning "Subagents launch in the background" in the same document describes the fan-out as intended. The description says both documents already describe this behavior and the code now matches them; that paragraph is the one place it doesn't. Since the audit reads this document as its spec, it's a live source of a spurious FAIL. Happy to push the correction onto this branch or leave it for its own PR.
  • The timeout comment says the relationship is written down "so a future edit to either number doesn't silently re-break it", but only the workflow side carries it. DEADLINE=$(( $(date +%s) + 1500 )) in .github/audit/orchestrator.md has no reciprocal note, so raising the deadline there re-breaks the invariant in exactly the direction the comment is guarding against.

Minor: the three fragment filenames are now spelled out in four places in this file — the redactor's argv, the fail-closed rm -f, the artifact path:, and the guard loop — and drift between two such lists is precisely what the FAIL IF on the redactor exists to catch. A job-level env: holding the list would cover three of the four.

Comment thread .github/workflows/security-audit.yaml Outdated
… one fragment list, correct the claude_args paragraph
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 26, 2026

Copy link
Copy Markdown

Deploying mouseterm with  Cloudflare Pages  Cloudflare Pages

Latest commit: fa5bf1d
Status: ✅  Deploy successful!
Preview URL: https://cd926975.mouseterm.pages.dev
Branch Preview URL: https://fix-audit-fragment-durabilit.mouseterm.pages.dev

View logs

@dormouse-bot

Copy link
Copy Markdown
Collaborator Author

All three of the outside-diff points are on the branch as of ed9d384; the inline suggestion is answered in thread.

SECURITY.md — taken here rather than left for its own PR. The paragraph is right that this is a live source of a spurious FAIL: the audit reads this document as its spec, and the sentence tells it that a config it can see with its own eyes is a violation. It also isn't really outside this PR's scope — the description asserted "both already describe this behavior, and the code now matches them", and that sentence is the counterexample to the claim the PR makes about itself, so shipping the two apart would leave the description wrong in the interval. The rewrite says what the config does (Task/Agent allowed, Workflow denied) and names the control that actually holds — the orchestrator's non-turn-ending wait, and the FAIL IF requiring it — since --allowed-tools only auto-approves and so was never what prevented the failure. That matches the "Subagents launch in the background" paragraph two above, which already said as much.

The timeout note is reciprocal now. The workflow-side comment was doing all the work, which made it a one-way guard against a two-way edit. .github/audit/orchestrator.md carries the other half above DEADLINE=$(( $(date +%s) + 1500 )), naming timeout-minutes: 40 and what re-breaks if the deadline is raised alone.

The four lists are three. A job-level AUDIT_FRAGMENTS now feeds the redactor's argv, its fail-closed rm -f, and the guard loop. The artifact path: is the one that can't consume it — upload-artifact wants one entry per line, and interpolating a multi-line env var into a block scalar is a cleverness I'd rather not have between an accidental printenv and a public artifact — so it stays spelled out with a pointer back to the env var. Three-of-four, as you had it.

Verification, and one thing worth recording

Same fixture approach as the original PR — the two shell steps were extracted from the shipped YAML by parsing it, not retyped, so the tested text is the text that runs.

Redactor, happy path (fake secret values planted in the transcript and two fragments), confirming the env var expands to the same three files the old literal list named:

::warning::Redacted 3 literal secret occurrence(s) from agent output.
step exit: 0
claude-execution-output.json:    {"leak":"***"}
audit-report.md:                 report body, clean
audit-supply-chain.md:           supply fragment leaked *** here
audit-ci-secrets.md:             ci fragment leaked *** here
audit-application.md:            appsec fragment clean

Fail-closed, with one fragment made unreadable so the redactor throws partway through: exit 1, and the transcript, the report, and all three fragments are gone rather than left on disk for the if: always() archive step.

Worth recording because it recurred: writing the AUDIT_FRAGMENTS rationale as a shell comment put it inside the single-quoted node -e script on the first attempt, where Node parses it as JavaScript and the step fails closed on every run — the same trap the original PR description flagged. bash -n passes it, because it is all inside a quoted string. The fixture run is what caught it, again. The comment block above node -e now says to keep shell comments outside the quotes and why.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four findings from the previous review are applied and check out — the plural MISSING_FRAGMENTS body, the single AUDIT_FRAGMENTS list feeding three of the four sinks, the reciprocal timeout note, and the claude_args paragraph in SECURITY.md now matching the shipped --allowed-tools "…,Task,Agent" --disallowed-tools "Workflow". One new finding, on the premise of the timeout change itself.

The 25-minute deadline still cannot fire, so timeout-minutes: 40 is not the missing half of the invariant it is described as. Two things stack: the deadline is 1500s while a single Bash call is capped at ten minutes (§2 of orchestrator.md says so itself), so [ "$(date +%s)" -ge "$DEADLINE" ] is never true inside any one call; and §2 then instructs "re-issue the same loop", where the same loop re-runs DEADLINE=$(( $(date +%s) + 1500 )) in a fresh shell, pushing the deadline out another 25 minutes each time. echo "DEADLINE"; break is unreachable on both counts, and the only thing actually bounding the wait is still the runner's cancellation — which this PR moves from 20 minutes to 40.

That inverts what the two new comments claim. The workflow-side one says the graceful path "was unreachable" at 20 minutes; it is unreachable at 40 too, just later and at roughly double the spend — and this is the wait that cost $5 and produced no verdict on run 32618922852, so a stalled run will use the whole budget rather than giving up at 25 minutes with the fragments it has. It also leaves SECURITY.md's FAIL IF the orchestrator prompt stops requiring … a bounded deadline satisfied only in form: the bound is written down but never binds.

Inline suggestion persists the deadline across re-issues, which makes it fire on the third one (at ~25 minutes of accumulated wait) and makes the timeout relationship the comments describe real in both directions. The workflow-side comment reads correctly once that lands, so it needs no separate edit.

Nothing else in ed9d384 — the guard loop, the fail-closed rm -f over $AUDIT_FRAGMENTS, and the two-cause INCONCLUSIVE branch all trace clean, including the PASS + missing-AUDIT_PAT and FAIL + empty-report paths, which correctly skip the new guard.

Comment thread .github/audit/orchestrator.md Outdated
The 25-minute deadline was recomputed from `now` in every shell that ran
the wait loop, and 1500s exceeds the ten-minute Bash cap, so the
`DEADLINE` branch was unreachable twice over: never within one call, and
pushed out another 25 minutes by each re-issue the prompt itself asks
for. `timeout-minutes: 40` therefore bought a longer stall, not a
graceful give-up.

The deadline now lives in `${RUNNER_TEMP:-.}/audit-deadline`, written
once and read back by every re-issue, so the 25 minutes accumulate and
the branch fires on the third call. SECURITY.md's `FAIL IF` names the
persistence as part of the bound, since a recomputed deadline satisfies
"bounded" in form while leaving the runner's cancellation as the only
thing ending the wait.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deadline now persists and the arithmetic works — two successive reads of $DEADLINE_FILE give the same value, and [ "$(date +%s)" -ge "$DEADLINE" ] becomes true partway through the third call. The reciprocal comments, the SECURITY.md FAIL IF, and the .gitignore entry all follow from that correctly. Three things the newly-reachable path exposes.

The graceful give-up lands as FAIL, not INCONCLUSIVE. The workflow comment this PR adds says that at 20 minutes "every overrun landed as INCONCLUSIVE — the exact failure mode the deadline exists to prevent". What an overrun lands as now is worse on the same axis. When the DEADLINE branch fires, the orchestrator breaks out with a fragment missing, and §4 of .github/audit/orchestrator.md tells it to "FAIL if any subagent returned FAIL, or if any of the three fragments is missing or empty" — so it writes FAIL to audit-status.txt. The reporting step's else branch then files [security-audit] FAIL on <date> with the body "Audit failed at …", and the upward-only relabel edits an existing open issue's title up to FAIL too. A run that simply ran out of time is published as a security finding, which is the conflation the three outcomes exist to prevent — and the guard added in this same PR says so in its own comment: "Downgrade to MISSING rather than FAIL for the same reason the three outcomes exist above — this is an audit that did not finish, not a security finding." The guard only catches PASS with a missing fragment; the prompt routes the identical condition to FAIL.

SECURITY.md takes the guard's side, in the paragraph beginning "The audit runs as a single headless turn": audit-status.txt "is written only once the verdict covers every check". A deadline-expired run has not covered every check, so on that document's own terms it should write no status file at all and let the existing MISSING branch reproduce the partial report. The narrow fix is to split §4's or: a subagent that returned FAIL is a verdict, a missing fragment with no FAIL from any domain is not. I'd want your read before changing it — it moves what the audit publishes, and §4's closing line ("The reporting step checks this independently … but do not make it do that work") reads as if the FAIL were deliberate.

The ten-minute cap is the ceiling, not the default. §2 says "A single Bash call is capped at ten minutes" and the new prose derives "fires on the third one" from it, but 600000 ms is the tool's maximum; the default is 120000. Nothing sets it otherwise — the action step passes only --allowed-tools/--disallowed-tools, and no BASH_DEFAULT_TIMEOUT_MS is in the job env — so unless the orchestrator asks for the maximum, each wait returns after two minutes and 25 minutes takes about thirteen re-issues. The wall-clock bound still holds; what changes is a dozen agent round-trips instead of two, and an agent seeing its "ten-minute" wait return in two minutes has more reason to read that as the subagents having died. Inline suggestion asks for the timeout explicitly (and corrects "third one" → "third call" while it's there — counted as re-issues rather than calls, the branch fires on the second).

Nothing clears audit-deadline on the ${RUNNER_TEMP:-.} fallback path. In CI RUNNER_TEMP is fresh per job, so the file is always new. Off-runner it lands in the repo root — which is why the .gitignore entry is here — and it survives, already expired, into the next run: the second invocation reads a past deadline and breaks out with DEADLINE on the loop's first iteration, before any subagent could have written anything. scripts/security-audit-local.sh doesn't run the orchestrator today, so this is only reachable by running the prompt by hand. Given that, dropping the fallback (and the .gitignore entry with it) is probably tidier than clearing the file — the orchestrator only ever runs in CI, and the fallback's sole effect is a stale file that silently defeats the loop.

Comment thread .github/audit/orchestrator.md Outdated
The wait deadline is now reachable, and §4 of the orchestrator prompt sent
what it produces — a missing fragment — to `FAIL`, which files a run that
ran out of time as `[security-audit] FAIL` and relabels an open issue up to
it. Split the `or`: a subagent that returned `FAIL` is a verdict, a missing
fragment with no domain failing is not, and the prompt now writes no status
file there so the reporting step's INCONCLUSIVE branch reproduces the
partial report instead.

Also: ask for the maximum Bash timeout in the wait call (600000 ms — the
harness default is two minutes, so the 25 minutes would otherwise take a
dozen re-issues rather than three), and drop the `${RUNNER_TEMP:-.}`
fallback, whose only effect off-runner was a stale deadline file that broke
the loop on its first iteration.
@dormouse-bot

Copy link
Copy Markdown
Collaborator Author

All three are on the branch as of 4286a15e; the inline suggestion is answered in thread.

The FAIL-vs-INCONCLUSIVE split — taken, and it's the load-bearing one. The review asked for a read before changing it, on the grounds that §4's closing line reads as if the FAIL were deliberate. I don't think it was: that line is about not making the reporting step do the orchestrator's work on the PASS path, and the same PR that adds the PASS-with-missing-fragment guard argues in the guard's own comment that this condition is "an audit that did not finish, not a security finding". SECURITY.md says the same thing in the paragraph beginning "The audit runs as a single headless turn". The FAIL was the one place that disagreed, and it only started mattering once this PR made the deadline reachable — before it, the give-up path never fired, so nothing was routed through it. §4 now splits the or: a subagent that returned FAIL is a verdict and stays FAIL regardless of what the other domains did; a missing fragment with no domain failing writes no status file, which lands on the existing MISSING branch and reproduces the partial report. Both still exit non-zero and hold the release gate shut, so nothing loosens — what changes is which issue a stalled run files.

Two consequences worth naming, since this moves what the audit publishes: the FAIL IF at SECURITY.md:274 said "must fail the audit, not be absent from it", which the new routing contradicts on its face — it now says must not pass, and states the other direction explicitly so the invariant survives the next edit in both directions. And the guard's comment in the workflow no longer claims the prompt is told to FAIL on a missing fragment, because it isn't.

The stale deadline file. Dropped the ${RUNNER_TEMP:-.} fallback and the .gitignore entry with it, per the suggestion — RUNNER_TEMP is always set in Actions and fresh per job, scripts/security-audit-local.sh runs the domains directly and never the orchestrator, and the fallback's only reachable effect was handing an expired deadline to the next hand-run.

@nedtwigg — the first item is the one to look at if you look at one. It's a behavior change to what the nightly publishes, decided by the author on the reviewer's reasoning rather than by you.

Why the deadline branch reached FAIL at all

The chain: the DEADLINE branch breaks out of the wait with a fragment missing → §4 said "FAIL if any subagent returned FAIL, or if any of the three fragments is missing or empty" → FAIL in audit-status.txt → the reporting step's else branch titles the issue [security-audit] FAIL on <date> with the body "Audit failed at …" → and on an existing open issue the upward-only relabel edits its title up to FAIL too. The PASS-with-missing-fragment guard added in this PR never sees it, because it is gated on STATUS = PASS.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deadline split traces clean — FAIL from a subagent stays FAIL regardless of what the other domains did, a missing fragment with no domain failure writes nothing and lands on the existing MISSING branch, both still exit 1 and hold the release gate. The RUNNER_TEMP fallback removal checks out too: scripts/security-audit-local.sh verifies orchestrator.md exists but never runs it, so CI really is the only caller. Two findings, both about what a reader of the resulting issue gets.

§3 still forces a binary Summary, so the INCONCLUSIVE issue reproduces a report that contradicts its own title. §4 now has three outcomes — FAIL, PASS, and write-no-status-file — but §3 above it is unchanged: "append a ## Summary section: overall PASS or FAIL, a one-paragraph rationale, and one line per domain giving that domain's verdict." §3 runs first, by §4's own closing paragraph ("Write audit-report.md before audit-status.txt, always"), so on the deadline path the orchestrator is told to commit to PASS or FAIL in the report body and then withhold the status file. The MISSING branch cats that report into the issue, so the published body reads "Audit produced no verdict at …" followed by a ## Summary asserting one — and if the agent picks PASS (two domains reported clean, the third never ran), a public security-audit issue carries an overall PASS over an unaudited domain, which is the shape the FAIL IF on line 274 exists to prevent. Before this commit the two agreed, because §4 routed the same condition to FAIL. Smallest fix is to give §3 the third word — "overall PASS, FAIL, or INCONCLUSIVE, the last when §4 tells you to write no status file" — which is outside the diff, so I've left it here rather than as a suggestion. Happy to push it.

The prose the deadline path now lands on points at a marker that won't be there. The no-MISSING_FRAGMENTS branch tells the reader "the areas it never reached are marked UNVERIFIABLE below — start there rather than in the transcript". That was written for the turn-ending failure (runs 31927560706 / 32100728239), where the agent was mid-audit and UNVERIFIABLE is what _preamble.md asks a subagent to write for a check it could not determine. A deadline-expired run is a different shape: the domain wrote no fragment at all, so §3's merge emits _No report — this domain produced no fragment._ and there is no UNVERIFIABLE anywhere. Since this PR makes the deadline path reachable and routes it here, that is now the likelier of the two causes, and the reader greps for the wrong string. The enclosing comment's "no status file at all means start in the transcript" is wrong for it too — the report names the domain outright. Inline suggestion covers the prose; the comment above the if wants the same correction.

Minor: §4's "reproduces the partial report you wrote in §3, placeholders and all" holds for a missing fragment but not for the "or empty" case named in the same sentence. §3's merge is cat audit-supply-chain.md 2>/dev/null || echo "_No report …_", and cat on an existing zero-byte file succeeds, so the placeholder is skipped and that domain renders as a heading with a blank body. The two are reachable together — [ -s … ] in the wait loop is exactly what lets an empty fragment run to the deadline.

Comment thread .github/workflows/security-audit.yaml Outdated
…ader at the marker that's there

§3 forced a binary Summary while §4 now has three outcomes, so the
deadline path was told to commit to PASS or FAIL in the report body and
then withhold the status file — publishing an overall PASS over a domain
that never reported. §3 takes INCONCLUSIVE as a third word, and its
merge switches to [ -s ] so a zero-byte fragment gets the placeholder
instead of rendering as a heading with a blank body.

The INCONCLUSIVE issue's prose sent the reader to UNVERIFIABLE, which is
the subagent-level marker for a check the agent reached but could not
determine; a deadline-expired domain leaves _No report …_ instead. Name
both, in the issue body, in the comment above it, and in SECURITY.md.
@dormouse-bot

Copy link
Copy Markdown
Collaborator Author

All three are on the branch as of 61234062; the inline suggestion is answered in thread.

§3's binary Summary — taken, and it's the one that could have published something wrong. §4 grew a third outcome in the last commit and §3 above it didn't, so on the deadline path the orchestrator was told to commit to PASS or FAIL in the report body and then withhold the status file — and §4's own closing paragraph orders §3 first, so that's the sequence it actually runs. §3 now takes INCONCLUSIVE as a third word, tied to the condition rather than left to judgment ("the last whenever §4 below tells you to write no status file"), and says why the binary is wrong here: the issue reproducing that report is titled as having reached no verdict, so a ## Summary asserting PASS contradicts the issue carrying it and publishes an overall pass over an unaudited domain — the shape SECURITY.md:274 exists to prevent.

The "or empty" case is fixed at the merge rather than in the sentence. §4 claims the INCONCLUSIVE issue reproduces the partial report "placeholders and all"; the review is right that cat on a zero-byte fragment succeeds, so that domain rendered as a heading with a blank body and the claim held only for the missing case. Rather than qualify the sentence, §3's merge now tests [ -s ] — the same emptiness test the wait loop uses, which is what makes the two reachable together in the first place — so both cases land on the placeholder and §4's sentence is true as written. It also matters more now than the empty-file edge suggests: the prose that this PR routes the deadline path to tells the reader to look for _No report …_, and a blank body is exactly the state where they'd find nothing and conclude the domain reported clean.

The UNVERIFIABLE prose. Applied as suggested, plus the comment above the if (it said "no status file at all means start in the transcript", which is wrong for both causes now that the report names the domain outright). SECURITY.md had the same gap — "partial, with unreached areas marked UNVERIFIABLE" is only the turn-ending shape — so it names both shapes now, and mentions that the Summary may read INCONCLUSIVE, since it's the document §3 is written against.

Verification

Same fixture approach: the Surface result step and §3's merge block were extracted from the shipped YAML and the shipped prompt by parsing them, not retyped.

§3's merge on a mixed fixture — one fragment present, one zero-byte, one absent:

## Supply chain

supply chain body

## CI and secrets

_No report — this domain produced no fragment._

## Application security

_No report — this domain produced no fragment._

The zero-byte fragment is the middle one; under cat … || echo it contributed 0 bytes and no placeholder (checked directly, not inferred).

The reporting step, stubbed gh, both MISSING causes: the no-status-file path files [security-audit] INCONCLUSIVE, exit 1, with the corrected marker prose sitting above a body that contains the _No report …_ it points at; the downgraded-PASS path on a two-missing fixture still emits both ::warning:: lines, the plural body, and exit 1.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three findings from the previous review are applied and trace clean. §3's merge tests [ -s ] on all three fragments, so a zero-byte fragment lands on the placeholder and §4's "placeholders and all" is true as written rather than only for the missing case; §3's ## Summary takes INCONCLUSIVE tied to §4's write-no-status-file condition rather than to the agent's judgment; and the marker prose is the suggestion verbatim, with the enclosing comment corrected alongside it. SECURITY.md's two-shapes paragraph agrees with _preamble.md, which scopes UNVERIFIABLE to a check within a fragment that the subagent could not determine.

One thing the FAIL→INCONCLUSIVE reroute in 4286a15e left behind, outside the diff.

scripts/security-audit-local.sh still states the old CI contract. run_domain's failure path prints "==> $domain produced no fragment — in CI that is a FAIL" (the else arm of its [ -s "$out" ] check). That was accurate against the old §4 — "FAIL if any subagent returned FAIL, or if any of the three fragments is missing or empty" — but §4 now routes exactly that condition, a missing fragment with no domain returning FAIL, to no status file at all, and this PR's own FAIL IF edit in SECURITY.md says so in as many words: "It must not be published as FAIL either, unless some domain actually returned one." So the one place a maintainer iterating locally is told the CI consequence now names the outcome this PR exists to stop producing — and scripts/ is in the ci-and-secrets domain's scope, so the nightly reads both documents and can reasonably report the disagreement.

One line, e.g. — in CI that is an INCONCLUSIVE audit, not a FAIL. Happy to push it onto this branch.

Nothing else in 61234062 — the merge block, the three-outcome Summary, both INCONCLUSIVE branches, and the SECURITY.md paragraph all check out, including the downgraded-PASS path, which reaches its own prose and never the marker sentence.

…CONCLUSIVE

A domain that writes no fragment while no domain returned FAIL routes to no
status file, which the workflow reports as INCONCLUSIVE — the outcome this
branch exists to produce instead of a FAIL. The local runner still told a
maintainer iterating on a domain prompt that CI would call it a FAIL.
@dormouse-bot

Copy link
Copy Markdown
Collaborator Author

On the branch as of fa5bf1dd. run_domain's failure arm now reads — in CI that is an INCONCLUSIVE audit, not a FAIL, which matches §4 of orchestrator.md (no status file when a fragment is missing and no domain returned FAIL) and the FAIL IF bullet this PR edited in SECURITY.md. bash -n and shellcheck are clean.

That was the last open item — the review found nothing else in 61234062.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant