Skip to content

test(storage): wait for native-main startup before comparing the storage scan - #8819

Merged
lidge-jun merged 1 commit into
devfrom
fix/api-storage-coordination-race
Nov 16, 2026
Merged

lidge-jun merged 1 commit into
devfrom
fix/api-storage-coordination-race

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

tests/storage/api-storage.test.ts > "returns the scan report with buckets and totals, no error field" failed on windows 5/14 in dev batch run 43818098370 at f329892. The endpoint's scan reported {bytes: 666, fileCount: 4} and the inventory taken right after it {bytes: 1178, fileCount: 6}.

Cause. startServer() starts the native-main lifecycle, which acquires the CODEX_HOME owner lock and runs its first exclusive claim in the background after startServer() returns:

  • prepareOwnerDatabase creates the empty .opencodex-native-main.owner.sqlite, then awaits hardenStableLockFile. On Windows that hardening is an ACL subprocess (hardenSecretPathAsync).
  • The owner then holds BEGIN IMMEDIATE, which leaves .opencodex-native-main.owner.sqlite-journal at 512 bytes.
  • convergeOwnedStartup runs withNativeMainExclusiveClaim, which creates the empty .opencodex-native-main.claim.sqlite.

The scan saw the fixtures, config.toml (46 bytes) and the empty owner database. The inventory also saw the owner journal and the claim database, which is exactly +2 files and +512 bytes. Both files are in the test's allowed list. The product behaves as intended: this server syncs Codex, so it owns the native-main lock. Only the test raced that startup.

Fix. Before scanning, the test now awaits waitForNativeMainStartupGate(), the settle point other server tests already use. It resolves only after owner acquisition, recovery and the first claim. After it, the next CODEX_HOME writer is the 60-second stage sweep, and on a fresh home that sweep creates nothing. The test also asserts that the inventory taken after settling equals the one taken after the scan. Exact equality with body.total and the allowed coordination-file list are unchanged.

Not caused by this batch. In git log e792c9dce4..f329892401 -- src/codex src/server/index.ts src/storage the only commit is b378a43 (#8766). Of the owner, claim and lock-file code it changes only native-main-owner.ts, adding a publish() hook that ends retention on ownership loss. It creates no file and changes no timing. The awaited hardening that opens this window has existed since 6b9f734 (2026-08-06). The same test passed in the seven earlier failing dev batch runs I checked (43790630995 back to 43451726885), so this is the first observation. Inferred, not proven: the trigger is Windows runner timing.

Refs dev batch run 43818098370.

Verification

  • Reproduced in a temporary, uncommitted copy of the test. It slowed owner hardening by 150 ms through the existing nativeMainStartup.owner.hardenPath seam and paused 600 ms between the scan and the inventory, matching the CI timing.
    • Old ordering: failed 5/5 with the CI numbers (666/4 vs 1178/6).
    • New ordering: passed 5/5.
  • Linux (oven/bun:1.4.4 container): bun test tests/storage/api-storage.test.ts passed 20/20. A probe in the same container confirmed the gate settles ready with the owner database, owner journal and claim database present, so the owned path ran.
  • macOS: the same file passed 20/20 (run under /usr/bin/lockf -k /tmp/ocx-test.lock).
  • bun run typecheck: no type errors.
  • bun test tests/test-layout.test.ts tests/test-layout-tooling.test.ts passed, as did tests/ci-workflows/file-size-ratchet.test.ts and bun run structure:check.
  • Not run: the full suite (bun run test), because the local host is under heavy load. That coverage, and the Windows shard, is left to CI: ci:full on this PR runs the four Ubuntu shards, and Windows needs the next lane=all batch on dev.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. (Test-only change; none needed.)
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. (No auth or credential code changed.)

Summary by CodeRabbit

  • Tests
    • Updated storage API coverage to account for native startup completing before checking the stored-file inventory. The test also verifies that the inventory remains unchanged after a request, improving confidence in storage behavior during startup.

…age scan

GET /api/storage compares the endpoint's scan with a filesystem inventory
taken right after it. Native-main startup takes its CODEX_HOME owner lock
and runs its first exclusive claim in the background after startServer()
returns; on Windows the lock's ACL hardening is a subprocess. In dev batch
run 43818098370 the owner journal (512 bytes) and the empty claim database
landed between the scan (666 bytes, 4 files) and the inventory (1178 bytes,
6 files).

The test now awaits waitForNativeMainStartupGate(), which resolves only
after owner acquisition and the first claim, before scanning, and asserts
the tree did not change while the endpoint scanned it. Exact equality and
the allowed coordination-file list are unchanged.

Refs dev batch run 43818098370.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 16, 2026 07:18
@lidge-jun lidge-jun added the ci:full Run the four full Ubuntu test shards on this pull request label Nov 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T07:20:10.354621Z 927004d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature). label Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The storage API test now waits for native-main startup to settle before capturing the expected CODEX_HOME inventory. It compares that snapshot with the inventory after the request.

Changes: Storage API inventory check

Layer / File(s) Summary
Synchronize inventory assertion with startup
tests/storage/api-storage.test.ts
The test waits for waitForNativeMainStartupGate before capturing the expected CODEX_HOME inventory. It compares the post-request inventory with that snapshot.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Other

Merge Risk: Minimal · up to 92700

This change only adjusts a storage API test so it waits for native-main startup to settle before taking its inventory snapshot. It does not alter production behavior and addresses a reported flaky race. No concrete merge risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the test file scope (storage), the change type (test), and the core behavioral change (waiting for native-main startup before the storage scan). The raw summary and PR objectives confi…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the test file scope (storage), the change type (test), and the core behavioral change (waiting for native-main startup before the storage scan). The raw summary and PR objectives confirm the change adds a wait for waitForNativeMainStartupGate() before scanning and an unchanged-inventory assertion, matching the title. The title is specific, concise, and a single sentence, so it fully reflects the main change.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between f329892 and 927004d.

Files selected for processing: 1

  • tests/storage/api-storage.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

@lidge-jun

Copy link
Copy Markdown
Owner Author

Maintainer integration of #8819 at 927004d.

This repairs the windows 5/14 failure in dev batch run 43818098370 (f329892): tests/storage/api-storage.test.ts, where the scan saw 666 bytes in 4 files and the inventory saw 1178 bytes in 6.

  • Cause (read in the code, then reproduced): native-main startup finishes after startServer() returns. It creates the owner lock database, waits on ACL hardening (a subprocess on Windows), and then writes the owner journal (512 bytes) and the claim database. Those two files landed between the scan and the inventory. They are already on the test's allowed list, so the product is unchanged. The only merge in the batch that touches this code, fix(codex): retain the native-main login only through a rewrite (#7345) #8766, adds a hook and creates no files.
  • Fix: the test change is 9 lines. The test awaits waitForNativeMainStartupGate(), the settle point other server tests already use, and asserts that the tree did not change while the endpoint scanned it. The exact equality and the allowed-file list are unchanged.
  • Evidence:
    • With owner setup delayed through the existing hardenPath hook, the old ordering failed 5/5 with CI's exact numbers and the new ordering passed 5/5.
    • The real test passed 20/20 on Linux (bun 1.4.4 container, with native-main startup confirmed running) and 20/20 on macOS. Typecheck, the layout and ratchet tests, and structure:check passed.
  • CI on this exact head: all 31 checks passed, including test 1–5/5 (ci:full), gates and a successful enforce-target. assert-mergeable-review.sh --maintainer-integration passed.

The other red shard in that run, windows 13/14 (desktop-compatibility-trust, "real CurrentUser Root inspection gave no answer within 25000ms"), was already failing on dev before this batch. Dev batch run 43343080374 on 24024b4 (2026-11-13) failed the same shard with the same message, which is why #8631 added the recorder. No merge in the batch touches that path.

Next step: one batched lane=all on the resulting dev head.

@lidge-jun
lidge-jun merged commit 347ca0c into dev Nov 16, 2026
37 of 39 checks passed
@lidge-jun
lidge-jun deleted the fix/api-storage-coordination-race branch November 16, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Maintenance, CI, tests, refactors, or build changes (not a user-facing bug or feature). ci:full Run the four full Ubuntu test shards on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant