Repository navigation
fix(link): keep a multi-route link blocked while an unreadable pidfile or a reload may hide an earlier tunnel - #8616
Merged
Conversation
… and on a changed record At Home start the multi-route reaper read an unreadable pidfile as missing, so the link's first round started while an earlier ssh might still hold the port. It now uses the strict reader: an unreadable pidfile flags the link and warns once. A link blocked that way is read again every tick, so making the file readable turns the block into an ordinary watch and deleting it lifts the block without a restart. A reload that changed a flagged link's record cleared the flag without reading the pidfile; a new listener port changes every record and keeps every tunnel port. The link is now settled again from its pidfile, and the new watch only observes. Refs #8609
…its record, observe-only The reload re-check deleted the existing watch before reading the pidfile, so a pidfile that was deleted or replaced meanwhile lifted the block while the watched ssh could still hold the port. The watch now stays, re-keyed to the new instance. After a replacement it only observes: no signal and no pidfile removal, also when its processes end and when an unreadable pidfile becomes readable later. The guide no longer says a link starts as soon as its pidfile is readable again. Refs #8609
…op a removed link's block A watch read its pidfile leniently, so a file that turned unreadable while the watched process exited lifted the block, although it might name another tunnel. The watch now reads it strictly and hands an unreadable file to the per-tick re-read, keeping the block. Removing a link clears its flag and re-read state, and a link added by a reload is settled from its pidfile as at startup, so a re-added id is neither left blocked nor started past an earlier tunnel. Refs #8609
…hed watch handed it over A watch whose processes all ended while its pidfile was unreadable handed the file to the per-tick re-read with signalling allowed, so a replacement tunnel named by the file once readable could be sent SIGTERM and SIGKILL. To that watch any later file is a replacement, so the re-read now only observes it. Refs #8609
Contributor
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
|
✅ Deterministic PR hygiene checks passed. |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #8610 (issue #8609). A Remote Link with several routes must never start a route candidate while an earlier ssh tunnel of it may still run. That ssh may hold the Child's remote port, and the readiness proof relies on the candidate being the only tunnel. The Home supervisor enforces this with the
orphan-unverifiedflag and a watch on the process named in the link's pidfile. #8610 kept the flag when a flagged one-route link gained routes and its pidfile could not be read. The same gap remained on four other paths, and this PR closes them.At Home start an unreadable pidfile counted as missing. The multi-route branch of the startup reaper read the pidfile leniently, so a pidfile that existed but could not be read or parsed (a directory at that path, a permission error, garbage content) left the link unflagged, and its first round started at once. Startup now settles multi-route links through one helper,
recheckPidfile, which uses the strict reader from #8610. A missing pidfile (ENOENT) still means no earlier tunnel. A readable one is watched as before. An unreadable one flags the link and logs[link] pidfile unreadable; link stays blocked link=<id>once.A link blocked that way was never checked again. Links blocked by an unreadable pidfile are now read again on every 1 s supervisor tick. If the operator makes the file readable, the block becomes the ordinary watch, which waits for the ssh it names to exit. If the operator deletes the file once no ssh for the link is running, the block lifts on the next tick and the first round starts, with no restart. A restart while the file is still unreadable flags the link again.
structure/remote-link.mdand the Remote Link guide's stale-tunnel bullet now describe this way out.A reload that changed a flagged link's record cleared the flag without reading anything. A record instance covers the alias, host-key fingerprint, API key id,
createdAt, tunnel port and the store-wide listener port. A new listener port therefore changes every link's instance while every tunnel port stays the same. Each flagged link lost its flag, and every multi-route link could start a candidate while its earlier ssh still held the Child's port. Now a flagged link that stays multi-route keeps its watch, with every process it already holds, re-keyed to the new instance. A flagged link with nothing watched is settled again from its pidfile. Both are observe-only from then on, which keeps the existing rule that no watch acts for a replaced record: no signal, and no pidfile removal, also when the processes end and when an unreadable file becomes readable later.A running watch could end on an unreadable pidfile. The watch read its pidfile leniently. If the file turned unreadable and every watched process exited, the block lifted, although the file might now name another tunnel. The watch now reads strictly. In that case it hands the link to the per-tick re-read instead of lifting, and whatever the file names once readable is only observed, as for any replacement.
Removal and re-adding. Removing a link now clears its flag and re-read state with its watch. A link that a reload adds to this process is settled from its pidfile as at startup, so a link removed and added again with the same id is neither left blocked nor started past an earlier tunnel.
Comment. The
READY_PROOF_MScomment insrc/link/route-rounds.tsand the matching sentence instructure/remote-link.mdnow say that the first supervisor tick at or after a candidate's deadline stops it.What keeps this inert. Everything here applies only to links with several routes, which exist only after an operator adds a route with
ocx link routes add. For those links, behaviour changes only when a pidfile exists but cannot be read, when a flagged link's record changes or is removed and added again in a reload, or when a link is added by a reload (one extra pidfile read; a new link has no pidfile). One-route links take the old path, and no config key or default changes.Verification
Tests written first, all in
tests/clients/link-route-supervisor.test.ts. The harness gained a capture of the supervisor's warnings, and the pidfile cases use the supervisor's own reader on amkdtempdirectory. Every test below fails whensrc/link/supervisor.tsis put back todev(36ee226); I ran the final test file against that file: 48 pass, 13 fail.a multi-route link whose pidfile is a directory / holds no pidfile at start is blocked, warned once, and starts once the file is gone: ondev,orphanis undefined at start.an unreadable pidfile that becomes readable is watched like any earlier tunnel until its process is gone.a flagged one-route link that gains routes while its pidfile is a directory / holds no pidfile(the fix(link): cap a late grant at the first deadline and keep the orphan flag for an unreadable pidfile #8610 test, extended): ondev, deleting the file never lifts the block.a reload that changes a flagged link's record keeps it blocked while the earlier tunnel may live(a listener-port change): ondev, the flag is gone after the reload.a reload that changes the record of a link blocked by an unreadable pidfile keeps it blocked.a reload that changes a watched link's record keeps watching the earlier process after its pidfile is gone.after a reload that replaced its record, a watch only observes: no signal, and no pidfile removal when the process ends: ondevit hits the 5 s test timeout.a pidfile unreadable at a reload that replaced the record is never acted on once it becomes readable.a watch whose pidfile turns unreadable does not lift the block when its process ends.a pidfile replaced while unreadable, after the watched process ended, is only observed once readable: on the previous commit of this branch it sent SIGTERM and SIGKILL to the replacement.a blocked link that is removed and added again with the same id is settled from its pidfile, not left blocked.The existing
an orphan watch whose link was replaced sends no further signal and touches no pidfilestill passes. It is what made the reload path observe-only.Commands on the head (exit codes):
TMPDIR=/tmp bun test tests/clients/link-route-supervisor.test.ts tests/clients/link-supervisor.test.ts: 97 pass, 0 fail (exit 0).bun run typecheck: exit 0.bun run privacy:scan: exit 0.bun run structure:check: exit 0.TMPDIR=/tmp bun scripts/test.ts --parallel=2 tests/clients tests/server/link-management-routes.test.ts tests/test-layout.test.ts tests/test-layout-tooling.test.ts tests/ci-workflows/structure-ssot.test.ts tests/ci-workflows/ci-structure-gate.test.ts tests/ci-workflows/file-size-ratchet.test.ts(204 files, includingtests/clients/link-boundary.test.ts): 3468 pass, 67 skip, 3 fail (exit 1). The 3 failures are the Rust ocx-child tests (tests/clients/link-ocx-child-e2e.test.ts, two cases, and the Rust case oftests/clients/link-ocx-child-conformance.test.ts). They stop before any test code runs, becausecargoon the build machine has no default rustup toolchain. This branch touches no Rust code.NOT RUN: the full
bun run testsuite (the build machine allows focused files only; CI covers the rest); the three Rust ocx-child tests above (no toolchain); the docs-site build (one sentence changed in the English guide, which has no translated copies); Windows and macOS (same code; without a readable start time a watched process is never signalled anyway, as before).Review: four read-only rounds by one independent reviewer (gpt-6.1-terra, xhigh effort), continued across rounds.
Each finding has a test in the list above. Round 4 found no P0–P2 and no regression, and ran the two supervisor test files on the head (97 pass).
Checklist
Implemented by Claude Opus 5.5 (anthropic/claude-opus-5-5) in GOAT through the Codex harness; in-lane review by GPT-6.1 Terra (xhigh). Received as a git bundle from the VM that built it and pushed by the coordinator.