Skip to content

ci: allow 18 minutes for measured full Ubuntu suites - #8940

Closed
luvs01 wants to merge 4 commits into
lidge-jun:devfrom
luvs01:ci/refresh-measured-test-durations-20261119
Closed

luvs01 wants to merge 4 commits into
lidge-jun:devfrom
luvs01:ci/refresh-measured-test-durations-20261119

Conversation

@luvs01

@luvs01 luvs01 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The full Ubuntu lane exposed two separate budgeting problems: recently added files were underweighted, and the 15-minute outer job deadline still cut off healthy full-suite progress after the measured table refresh. In run 44462803582, two shards finished every file with zero failures immediately before cancellation; the one permitted retry still cancelled shard 4 after 655 of 672 files, with no failed assertion.

The measured-table refresh from the earlier commits has now been incorporated into dev 676baf650c2d2b020439341f65e1ccc57c5d3e90. Relative to that current base, this PR changes the full Ubuntu jobs.test.timeout-minutes budget from 15 to 18 minutes, updates its strict equality test, and appends the matching exact-blob approval snapshot. It keeps five shards, the scheduling algorithm, complete file selection, the 60-second per-test limit, ordinary 120-second batch deadlines, existing serial overrides up to 300 seconds, failure handling, and aggregate required-job checks. PR fast-lane selection, triggers, runner selection, permissions, macOS/Windows execution policy, and all other job budgets are unchanged.

The already integrated table uses 19 hosted Ubuntu/macOS job logs and the existing parser, averaging, merge and rendering functions. It contains 3,348 paths. All 20 rows that dev 80e8952357d11ccf1421b4aa5620e781625dcdb0 added or updated remain exact, including split Link stop tests; their 71 overlapping older samples are excluded. The concurrent dev changes were preserved by a normal merge. The current head has parents 146909fa3e2c1b5e1bc93d1fec029d54f5ce2367 and 676baf650c2d2b020439341f65e1ccc57c5d3e90; its delta against current dev is exactly the three budget/test/snapshot files. The 69 Node tests, 251 CI-contract tests and type check below were rerun on that integrated tree.

Release-evidence approval records are appended through the documented exact-blob procedure. The initial refresh retained all 26 historical records and appended the TSV fingerprint. This follow-up retains all 27 records and appends one snapshot that changes only the CI workflow fingerprint; its other 26 pins, policy version and required platform job counts remain identical. No release qualification rule or assertion is removed.

Why an 18-minute outer budget

Four cancellations in the same-source #840 run 44463273114 still had 18–65 files left after about 912 seconds. Those files cost 12.575–53.402 seconds in the preceding successful test run. Applying the median slowdown of matching completed files estimates complete times of 926–981 seconds (15.4–16.35 minutes). An 18-minute bound leaves about 99–154 seconds beyond that estimate for runner variation and finalization. These are extrapolations, not observed successful completions.

The additional allocation is bounded at three minutes per full Ubuntu shard. Independent process deadlines continue to stop hung batches. A second timing-only reallocation was not adopted: held-out observations did not show a consistent improvement. This follow-up retains the already reviewed timing table.

Verification

Current head — 82ee1a5a483c3e4e7756d5cb000f72ab98e376a5

  • Strict workflow and lane/batch contracts: 251 passed / 0 failed / 9,573 assertions across ci-workflows.test.ts, ci-review-lanes.test.ts, ci-shard-balance.test.ts, ci-crash-disposition.test.ts, and ci-test-lane.test.ts. Executed with the normal scripts/test.ts --no-env-file --no-install --isolate --parallel=1 ... --timeout 30000 wrapper in a private temporary namespace.
  • The CI-equivalent five Node test files passed 69 tests / 0 failures / 0 skips. node .github/scripts/release-evidence.cjs policy-check passed. Before the snapshot update, the production-pin test failed specifically for the changed workflow fingerprint; the test itself was not modified.
  • bun check --no-script --noEmit: 2,630 files, no errors. git diff --check: passed.
  • Parsed workflow comparison verified that jobs.test.timeout-minutes is the only semantic YAML change. The strict equality assertion now expects exactly 18. The original runner, parser, selection helpers, full test manifest, timing table and Windows table are unchanged.
  • Independent security review checked the exact three-file patch, the complete workflow, historical approval-record preservation and the unchanged trust/gate boundaries. Workflow blob: da0b271b0d8c362a3efe98485ebfc543f64b03a2; policy blob: 7df235ec7904dd19a4b2e588ecfba1da627e9972.
  • Hosted status for this exact head: Run 44472741389 has been created and is queued. The GitHub merge commit is 91e18f2f58e780b45e23267ec1680e302e841cff; its tree 366c68e852b6b96993c260161a9da9b5ebe6af4f exactly matches the locally validated integration tree. This is not a passing hosted result.

The full local runtime suite was not repeated for this workflow metadata change during concurrent validation work. The repository's focused-validation exception applies; exact-head hosted CI remains required before merge. The local checks and the historical measurements below are not a hosted pass for this commit.

Source evidence

  • Green dev lane=all run 44386662799, commit c38d6eef7986b3b9acbb6e8ac17407ab84e5633e: five Ubuntu and four macOS suite jobs.
  • Fork #840 run 44448708478, PR head 374867f6a068c50beaa2ec4e17e57876c88c87cc: completed primary-file samples only. The workflow was cancelled; shard 2/5's unfinished file is excluded, and shard 3/5 finished its tests before job-level cancellation. This run is not passing-suite evidence.
  • Fork #841 run 44448420609, PR head 31e5078742f928592d7f88287cdb98a688cbc975: the successful latest-attempt job set, including retained earlier successful jobs. This is PR Ubuntu evidence, separate from the cross-platform dev dispatch above.

The generated header records the source job IDs, cancelled-run limitation, and preserved upstream provenance. Original log hashes and sample attribution were verified. Independent review confirmed all 20 retained values match dev exactly, and every other row and the complete path set match the existing official CLI's unfiltered refresh output. Final TSV SHA-256: 2d56880fe64338d36b424bb73178b025078ca1482da5dd4be6e73c6792759be5; Git blob: a07f90e6098082e1452ea5cc075c801271321eca.

Historical local validation of the initial table refresh

  • bun test tests/ci-workflows/ci-shard-balance.test.ts tests/ci-workflows/ci-crash-disposition.test.ts: 42 passed / 0 failed / 6,670 assertions, with the normal preload and network guard.
  • bun run typecheck: 2,624 files, no errors.
  • node .github/scripts/release-evidence.cjs policy-check: passed. The CI command node --test .github/scripts/release-dispatch-guard.test.cjs .github/scripts/release-evidence.test.cjs .github/scripts/release-history.test.cjs .github/scripts/release-resume.test.cjs .github/workflows/deploy-docs.test.cjs passed 69 tests / 0 failures / 0 skips, including historical policy and legacy-tree checks. The original production-pin test was reproduced failing against the preceding exact CI tree before the snapshot was added.
  • bun run privacy:scan and git diff --check: passed.
  • Independent verification ran the actual current-dev batch script with test execution replaced by argument capture. Before and after, it assigned the same 3,358 general test files exactly once across five shards, with no missing or duplicate files, sorted order within each shard, and at most 12 files per batch. This checks the actual runner's allocation and coverage; it does not execute the test assertions.
  • At head 146909fa3e2c1b5e1bc93d1fec029d54f5ce2367, independent security review confirmed the two-file delta relative to dev: the generated TSV and its appended exact-blob approval snapshot. At that earlier head, workflow, runner, parser/generator, Windows table, test files and assertions remained those of dev. No credentials or raw logs are committed. That historical policy blob was 51c1f898f2bdd69b7a012bd191ac51acb3c1779b; its validated tree was 2b6f996c5644515021e55304530cb960ce7e7f25.

The earlier table-only update used the repository's documented focused-validation exception because concurrent worktrees share limited resources. Its hosted result is recorded below; historical runs and the allocation probe are not a passing result for that head or the current candidate.

Hosted evidence before the outer-budget change — 2026-11-19

Exact head 146909fa3e2c1b5e1bc93d1fec029d54f5ce2367 did not pass. In run 44462803582, attempt 1, the release-contract gate and test shards 1/5 and 2/5 passed. Shards 3/5, 4/5 and 5/5 were cancelled at the unchanged 15-minute job limit. Independent log review found no failed assertion in those three jobs.

  • 4/5 and 5/5 completed all their files and 62/62 and 73/73 batches, reporting 0 fail, before cancellation only 0.351 / 0.293 seconds after the last summary. Their job status is still cancelled, not successful.
  • 3/5 completed 640/671 files and 62/65 batches, then stopped during batch 63. Its 31 incomplete files are not claimed as tested. Those files cost 26.129 seconds in the earlier successful fix: release Windows ACL temp-path memos #840 test run, but that is not an exact estimate of the unfinished work here.
  • Comparing the same 640 completed files with the earlier successful fix: release Windows ACL temp-path memos #840 jobs gives 743.922 → 849.373 seconds; among 611 files with meaningful timings, 445 were over 10% slower. This distributed slowdown and the two finalization-only cancellations support a bounded retry without changing the tests or job policy.

At 05:39 UTC, one failed-job-only retry was started for shards 3/5, 4/5, 5/5 and their dependent aggregate. Already successful jobs are retained. Attempt 2 has now completed: shards 3/5 and 5/5 passed, but 4/5 again reached the unchanged time limit. It completed 655/672 files and 60/62 batches, then stopped during batch 61, with no failed assertion. Its 17 incomplete files cost 6.990 seconds in attempt 1; that is historical context, not a pass for attempt 2. The aggregate CI check failed at 05:59:49 UTC. That earlier head remains red, and no further retry was started. A proposed second table reallocation was rejected after held-out observations regressed. The current follow-up instead changes the outer full-suite job budget to 18 minutes while preserving individual process deadlines and failure assertions.

Historical timing replay and limits

Using the retained 20 dev values and completed #840 measurements for the other files, with same-file #841 measurements filling cancellation gaps, the largest modeled Ubuntu shard load falls from 841.100 to 818.582 seconds relative to the then-current dev 80e8952. A replay over the previously observed macOS file set also improves its largest modeled load from 1,318.451 to 1,235.975 seconds.

These are allocation estimates, not new CI passes. They exclude setup costs, process-boundary effects and runner variation. The original refresh's general manifest had 17 paths without a timing row, which use the existing median fallback. Changed and new macOS files have not acquired current macOS execution evidence from this refresh. Required CI and maintainer review remain necessary before merge.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. Measurement provenance is recorded in the generated table; product behavior is unchanged.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. Independent review confirmed the measured scheduling inputs, the bounded outer-job change and preserved gates; no credentials or raw logs are committed.

Summary by CodeRabbit

  • Chores
    • Updated internal release-validation records with the Windows test shard configuration and file verification data.
    • Increased the Linux test job’s time limit from 15 to 18 minutes to allow additional time for setup and cleanup around test batches.
    • These changes affect CI and release checks only; no app features, behavior, or interface changes are included.

Regenerate the shared duration table from completed primary-file samples in hosted Ubuntu and macOS jobs. Preserve the existing shard count, deadlines, file selection, and scheduling algorithm.
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The test job timeout increases from 15 to 18 minutes. The workflow comments and test comments describe setup and cleanup around bounded batches. The release-evidence policy adds a CI record with 14 Windows shards and file hashes.

Changes: CI timeout and release evidence

Layer / File(s) Summary
Update test timeout and policy record
.github/workflows/ci.yml, tests/ci-workflows/ci-workflows.test.ts, .github/scripts/release-evidence-policy.json
The test job timeout and its test assertion change from 15 to 18 minutes. The Windows runner-selection timeout remains 2 minutes. The policy adds a CI record with 14 Windows shards and hashes for listed files, including the scripts/ci/test-durations.tsv hash a07f90e6098082e1452ea5cc075c801271321eca.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Possibly related PRs

Merge Risk: Minimal · up to 82ee1

The change gives the Linux CI test job three more minutes, refreshes the CI timing data, and records the policy fingerprints, which match the current files. No defects were found in the diff. Before merging, confirm that the required hosted CI passes on this exact head, because the author reports that an earlier head still timed out on one shard.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "ci: allow 18 minutes for measured full Ubuntu suites" matches the main change in the changeset: the test job timeout in .github/workflows/ci.yml rises from 15 to 18 minutes, and the matchin…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title "ci: allow 18 minutes for measured full Ubuntu suites" matches the main change in the changeset: the test job timeout in .github/workflows/ci.yml rises from 15 to 18 minutes, and the matching expectation in tests/ci-workflows/ci-workflows.test.ts is updated. The title is specific and concise, and it names the Ubuntu full-suite scope. The PR objectives describe a larger change (refreshed test durations in scripts/ci/test-durations.tsv and a release-evidence policy snapshot). The title does not mention those parts, but it does not contradict them and it does not need to cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 146909f and 82ee1a5.

Files selected for processing: 3

  • .github/scripts/release-evidence-policy.json
  • .github/workflows/ci.yml
  • tests/ci-workflows/ci-workflows.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Apply the identical generated timing table from lidge-jun#8940
(d190515).
Completed primary-file samples update scheduling weights only; test coverage,
five-shard execution, time limits, and workflow policy remain unchanged.
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Apply the identical generated timing table from lidge-jun#8940
(d190515).
Completed primary-file samples update scheduling weights only; test coverage,
five-shard execution, time limits, and workflow policy remain unchanged.
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Apply the identical generated timing table from lidge-jun#8940
(d190515).
Completed primary-file samples update scheduling weights only; test coverage,
five-shard execution, time limits, and workflow policy remain unchanged.
@github-actions github-actions Bot added the chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature). label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Preserve both histories and the existing stop evidence corrections.
Resolve only the measured-duration table conflict with the reviewed
upstream-preserving table from lidge-jun#8940
(e584532).
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Preserve upstream same-handle argv reads, Hub incarnation checks, relay
refresh cleanup, volatile lifecycle locks and fixture phase deadlines.
Keep strict native owner evidence, Windows argv parsing, rollback retry
ownership and Link recycle exclusion from this follow-up.

Use the reviewed upstream-preserving timing table from
lidge-jun#8940 (e584532).
Native Windows acceptance remains outstanding; keep this PR in draft.
@luvs01
luvs01 marked this pull request as ready for review November 19, 2026 04:51
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ❌ Failed 2026-11-19T04:51:32.306114Z
We couldn’t complete this review. Comment @codex security review to try again.
e584532 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins.

Preserve the five shards, per-test and batch deadlines, lane selection,
permissions, failure checks, and historical release evidence. Integrate
current dev and append the exact workflow fingerprint after independent
review of the three-file change.
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Preserve the existing PR work while merging current dev. Keep five shards, individual test and batch deadlines, required checks, triggers and release identity rules. Append the reviewed workflow fingerprint without changing prior policy records.

Refs lidge-jun#8940
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Preserve the existing PR work while merging current dev. Keep five shards, individual test and batch deadlines, required checks, triggers and release identity rules. Append the reviewed workflow fingerprint without changing prior policy records.

Refs lidge-jun#8940
luvs01 added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
Preserve the existing PR work while merging current dev. Keep five shards, individual test and batch deadlines, required checks, triggers and release identity rules. Append the reviewed workflow fingerprint without changing prior policy records.

Refs lidge-jun#8940
@luvs01 luvs01 changed the title ci: refresh measured test durations for balanced batches ci: allow 18 minutes for measured full Ubuntu suites Nov 19, 2026
lidge-jun added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
The 18-minute shard timeout is a CI policy change tracked separately in lidge-jun#8940; this PR keeps dev's .github/workflows/ci.yml and release-evidence policy unchanged.

Co-authored-by: luvs01 <luvs01@hanmail.net>
lidge-jun added a commit to luvs01/opencodex that referenced this pull request Nov 19, 2026
The 18-minute shard timeout is a CI policy change tracked separately in lidge-jun#8940; this PR keeps dev's .github/workflows/ci.yml and release-evidence policy unchanged.

Co-authored-by: luvs01 <luvs01@hanmail.net>
lidge-jun added a commit that referenced this pull request Nov 19, 2026
* fix(responses): scope keyless loopback replay to its WebSocket

Bind replay for loopback Responses clients without a context principal to
the admitted WebSocket and its handshake thread. Keep scoped entries in
bounded memory and release them when the socket closes. Preserve the
durable replay path for callers with a principal and keep scoped writes
isolated from snapshots and spill storage.

Carry dev's peer-local classification alongside the replay scope during
the upgrade.

Co-authored-by: luvs01 <luvs01@users.noreply.github.com>

* fix(responses): close replay admission and persistence review boundaries

Recheck scope lifetime before mutation, reject scoped spill intake, share credential-first replay ownership, and refuse missing strict replay IDs before upstream dispatch. Preserve legacy undefined-owner behavior and durable entries across cold snapshot errors and declined scoped writes.

Strengthen persistence, credential, HTTP, in-flight and fixture teardown regressions; document the strict continuation boundary in English, Korean and Spanish.

* fix(responses): preserve strict replay refusal and snapshot retention

Record zero-send native refusal attribution without entering recovery or
upstream dispatch. Skip malformed spill digests and reserve snapshot load
retries for filesystem failures.

Co-authored-by: luvs01 <luvs01@hanmail.net>

* fix(tests): include replay owner in Responses source inventory

Cover the shared credential-first replay owner in the existing module graph,
line-budget and source-oracle checks. Preserve all assertion and time limits.

* ci: keep dev's timing table in the replay-scope PR

Co-authored-by: luvs01 <luvs01@hanmail.net>

* ci: keep dev's workflow timeout out of this replay PR

The 18-minute shard timeout is a CI policy change tracked separately in #8940; this PR keeps dev's .github/workflows/ci.yml and release-evidence policy unchanged.

Co-authored-by: luvs01 <luvs01@hanmail.net>

---------

Co-authored-by: JUN <jun@lidgeai.com>
Co-authored-by: luvs01 <luvs01@users.noreply.github.com>
@luvs01

luvs01 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as superseded by #8944, merged in 24ae9b5. At dev e6432db, the full Ubuntu jobs.test timeout is 20 minutes, the job-specific regression test requires exactly 20, and the release-evidence snapshot pins that workflow blob (4ae44e193c6813a0378700f37c1cebaca7327cf1). Applying this PR's remaining three-file change would reduce the accepted budget to 18 minutes.

The measured scripts/ci/test-durations.tsv is already on dev. Its Git blob is a07f90e6098082e1452ea5cc075c801271321eca on dev, on this PR's head 82ee1a5a483c3e4e7756d5cb000f72ab98e376a5, and on the fork PR luvs01#840 at 48a43573170eb6df3d0f0d9f0f28f90ed6e26a3d. The separate Remote Link changes in the fork PR remain open for review.

@luvs01 luvs01 closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant