Skip to content

fix(test-lock): finish a decided stale retirement through transient rename refusals - #8823

Merged
lidge-jun merged 1 commit into
devfrom
fix/test-run-lock-generation-barrier
Nov 16, 2026
Merged

lidge-jun merged 1 commit into
devfrom
fix/test-run-lock-generation-barrier

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Fix a latent race in the Bun test-run lock that failed generation gate prevents duplicate acquisition during stale verdict on Windows (dev batch run 43832690786, windows 9/14, job 133181709470: Private barrier timed out: A.acquired).

Stale-lock retirement swallowed an EACCES/EPERM rename refusal and gave up the generation gate. The next waiter then retired the lock that the gate holder had already judged stale.

Root cause: child A held the generation gate while it decided the seeded lock was stale. When A then tried to move the stale lock directory aside, the rename was refused once. reclaimStaleLock treated EACCES/EPERM as "not now", returned false and dropped the gate. B, polling every 10 ms, entered the gate, retired the stale lock itself and acquired. A then waited behind a live owner (B). The fixture's cleanup wrote B.release, which let A acquire, and both children exited 0. The CI log matches this: both lockChildTerminal lines are printed by finish() after the barrier threw, and the worker's only exit-0 path writes A.acquired first.

Mutual exclusion held throughout. The holder of the stale verdict lost the retirement it had decided, and the test asserts that ordering. On Windows, a rename of a directory is refused while any process has a file inside it open. Antivirus can cause that, and so could the lock itself: every waiter read owner.json outside the gate on each 10 ms poll. In real CI this costs a queue slot or a poll cycle, and the retirement step gave no signal about it.

Summary

  • scripts/test-run-lock.ts: the quarantine rename now retries EACCES/EPERM inside the gate with backoff (5 ms to 100 ms) for up to 2 s while the generation is unchanged. If the rename is still refused after that, it defers as before. Other errors still throw.
  • The acquire loop reads owner.json outside the gate only to announce the wait or build the timeout message, instead of on every poll.
  • tests/helpers/test-run-lock-generation-worker.ts: the worker traces each stage to stderr, and the fixture prints that with the exit code. An A or B worker that returns without its own generation now throws instead of exiting 0. A private fixture file enables a seam that refuses renames of the lock path for a time window.
  • tests/ci-workflows/test-run-lock-generation.test.ts: a new case, stale verdict through a transient rename refusal, refuses A's rename for 300 ms. A barrier timeout now lists the barrier files present, so B.acquired would show up directly. The generation-gate assertions are unchanged.
  • structure/ops/test-sandbox-cleanup.md documents both behaviors.

Verification

All test runs were single files under /usr/bin/lockf -k /tmp/ocx-test.lock on macOS. The host was under load, so no full suite was run.

  • Repro before the fix (product file stashed, new case only): fail, Private barrier timed out: A.acquired (present: A.blocked, A.refuse-rename-ms, A.resume, A.waited, B.acquired, B.waited). The traces show B acquired at 24 ms and A acquired at 20061 ms, with both exiting 0. This is the CI signature.
  • With the fix: bun test tests/ci-workflows/test-run-lock-generation.test.ts: 15 pass, 0 fail. The refusal case passes in 494 ms.
  • bun test tests/ci-workflows/test-runner.test.ts: 92 pass, 0 fail (3 skipped by the file).
  • bun test tests/ci-workflows/test-home-guard.test.ts: 27 pass, 0 fail.
  • file-size-ratchet.test.ts 9/9, tests/test-layout.test.ts 2/2, structure-ssot.test.ts 51/51.
  • bun run typecheck, bun run structure:check and bun run privacy:scan: passed.

NOT RUN: full bun run test (host load limit), and Windows. Windows evidence will come from the next lane=all batch on dev.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Summary by CodeRabbit

  • Bug Fixes

    • Lock acquisition now recovers from temporary filesystem permission errors when reclaiming stale locks, improving reliability under transient conditions.
    • Waiting for a lock avoids repeatedly reading its owner information during polling.
  • Documentation

    • Updated the lock documentation to describe retry behavior and when owner information is read.

…ename refusals

The generation-gate holder swallowed an EACCES/EPERM quarantine rename and gave up the gate, so the next waiter retired the lock it had judged stale. Retry the rename inside the gate for a bounded window, stop per-poll owner reads outside the gate, and make the generation worker trace its stages and refuse a silent exit 0.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 16, 2026 09:16
@lidge-jun lidge-jun added the ci:full Run the four full Ubuntu test shards on this pull request label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T09:18:38.104935Z 1deb84e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the bug Something isn't working label Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The lock recovery path now retries stale-directory renames refused with EACCES or EPERM. The wait loop limits reads of owner.json. Documentation and generation-gate tests cover these behaviors.

Changes: Test lock recovery

Layer / File(s) Summary
Stale lock recovery and waiter reads
scripts/test-run-lock.ts, structure/ops/test-sandbox-cleanup.md
reclaimStaleLock retries EACCES and EPERM rename failures with increasing delays, up to two seconds. It stops if the lock disappears, its generation changes, or the next delay would exceed the deadline. The wait loop reads owner.json before announcing a wait or after the timeout threshold, rather than on every poll. The documentation describes these behaviors.
Transient refusal test coverage
tests/helpers/test-run-lock-generation-worker.ts, tests/ci-workflows/test-run-lock-generation.test.ts
The worker can simulate a timed window of EPERM refusals and records structured lock events. The test adds a transient-refusal scenario and verifies that a later rename succeeds. Barrier timeout errors include the sorted names of present barriers. The worker also checks acquisition results against the expected role.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Possibly related PRs

  • lidge-jun/opencodex#7508: Introduced the SQLite generation gate that serializes stale retirement and owner publication used by this lock recovery path.

Suggested labels: bug

Merge Risk: Minimal · up to 1deb8

This fixes a Windows race in the test-run lock by retrying transient rename refusals while the lock is held. It affects only the test infrastructure, and no concrete merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: a stale-lock retirement that was already decided is finished by retrying the quarantine rename through transient EACCES/EPERM refusals. The raw summary confirms the re…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: a stale-lock retirement that was already decided is finished by retrying the quarantine rename through transient EACCES/EPERM refusals. The raw summary confirms the retry logic in reclaimStaleLock, the matching documentation, and the new test case, so the title is specific and related to the changeset. It is not vague, and it does not describe anything absent from the diff.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3c8e1e7 and 1deb84e.

Files selected for processing: 4

  • scripts/test-run-lock.ts
  • structure/ops/test-sandbox-cleanup.md
  • tests/ci-workflows/test-run-lock-generation.test.ts
  • tests/helpers/test-run-lock-generation-worker.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration of #8823 at 1deb84e.

This repairs the windows 9/14 failure in dev batch run 43832690786 (3c8e1e7): test-run-lock-generation.test.ts > "generation gate prevents duplicate acquisition during stale verdict" ("Private barrier timed out: A.acquired").

  • Cause: worker A held the coordination gate and reached a stale verdict. Its quarantine rename of the stale lock directory was refused once with EPERM/EACCES. On Windows that happens while any process holds a file inside the directory open, such as antivirus or a waiter reading owner.json on every 10 ms poll. reclaimStaleLock then gave up the gate at once, B retired the lock and acquired, and A waited behind B past the 20 s barrier. Two runs never held the lock together, but a legitimate run could lose its turn and wait longer.
  • Fix (scripts/test-run-lock.ts):
    • A refused rename is retried inside the gate with backoff (5 to 100 ms) for up to 2 s. Each retry first re-checks that the lock is the same stale generation; ENOENT still defers, other errors still throw, and a rename still refused after 2 s defers as before.
    • Waiters read owner.json outside the gate only to announce the wait or build the timeout message, not on every poll.
  • Diagnostics: the worker now logs each stage and fails if it exits without acquiring, and a barrier timeout lists the barrier files that exist. The generation-gate assertions are unchanged. structure/ops/test-sandbox-cleanup.md documents the behaviour.
  • Evidence: a worker hook refuses A's rename for 300 ms. Without the fix this reproduces CI's pattern: B acquires at 24 ms, A at 20,061 ms, and the barrier times out. With the fix all 15 cases pass, the new one in 494 ms. test-runner (92) and test-home-guard (27) pass, as do typecheck, structure:check, privacy:scan and the guards. The Windows cause is inferred from Windows rename semantics; the next lane=all is the Windows evidence.
  • CI on this exact head: all 31 checks passed, including test 1–5/5 (ci:full) and enforce-target. assert-mergeable-review: OK.

Next step: one batched lane=all on the resulting dev head (#8822 and #8823).

@lidge-jun
lidge-jun merged commit 858ea8d into dev Nov 16, 2026
37 of 39 checks passed
@lidge-jun
lidge-jun deleted the fix/test-run-lock-generation-barrier branch November 16, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci:full Run the four full Ubuntu test shards on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant