Skip to content

feat(evals): report stability across repeated release test runs - #777

Open
Kidz0419 wants to merge 1 commit into
Tencent:mainfrom
Kidz0419:codex/release-test-stability
Open

Kidz0419 wants to merge 1 commit into
Tencent:mainfrom
Kidz0419:codex/release-test-stability

Conversation

@Kidz0419

Copy link
Copy Markdown

When a release scenario passes twice in three trials, the aggregate report now marks it flaky and shows that it passed at least once (pass@k) but did not pass every time (pass^k). JSON and Markdown reports include these per-case results, the overall proportions of planned cases meeting each condition, and the flaky case list.

Multi-run aggregation requires distinct run directories with matching candidate, CLI entry, binary, and test plan identities. It checks each on-disk plan against its recorded digest and derives results from each run's ledger, using the gate verdict when present. The single-run benchmark and release gate keep their existing behavior.

Fixes #772.

Validation

  • Release audit tests: 27 passed, with loopback excluded from the environment proxy.
  • pnpm eval:browser:check: 48 tests passed, with 12 cases and 21 fixture routes validated.
  • Skill bundle validation, bilingual changelog synchronization, and git diff --check passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: report pass@k / pass^k and flaky scenarios across repeated release-test runs

1 participant