Skip to content

fix(link): keep a multi-route link blocked while an unreadable pidfile or a reload may hide an earlier tunnel - #8616

Merged
lidge-jun merged 4 commits into
devfrom
fix/link-orphan-unreadable-pidfile-startup
Nov 13, 2026
Merged

lidge-jun merged 4 commits into
devfrom
fix/link-orphan-unreadable-pidfile-startup

Conversation

@lidge-jun

Copy link
Copy Markdown
Owner

Summary

Follow-up to #8610 (issue #8609). A Remote Link with several routes must never start a route candidate while an earlier ssh tunnel of it may still run. That ssh may hold the Child's remote port, and the readiness proof relies on the candidate being the only tunnel. The Home supervisor enforces this with the orphan-unverified flag and a watch on the process named in the link's pidfile. #8610 kept the flag when a flagged one-route link gained routes and its pidfile could not be read. The same gap remained on four other paths, and this PR closes them.

At Home start an unreadable pidfile counted as missing. The multi-route branch of the startup reaper read the pidfile leniently, so a pidfile that existed but could not be read or parsed (a directory at that path, a permission error, garbage content) left the link unflagged, and its first round started at once. Startup now settles multi-route links through one helper, recheckPidfile, which uses the strict reader from #8610. A missing pidfile (ENOENT) still means no earlier tunnel. A readable one is watched as before. An unreadable one flags the link and logs [link] pidfile unreadable; link stays blocked link=<id> once.

A link blocked that way was never checked again. Links blocked by an unreadable pidfile are now read again on every 1 s supervisor tick. If the operator makes the file readable, the block becomes the ordinary watch, which waits for the ssh it names to exit. If the operator deletes the file once no ssh for the link is running, the block lifts on the next tick and the first round starts, with no restart. A restart while the file is still unreadable flags the link again. structure/remote-link.md and the Remote Link guide's stale-tunnel bullet now describe this way out.

A reload that changed a flagged link's record cleared the flag without reading anything. A record instance covers the alias, host-key fingerprint, API key id, createdAt, tunnel port and the store-wide listener port. A new listener port therefore changes every link's instance while every tunnel port stays the same. Each flagged link lost its flag, and every multi-route link could start a candidate while its earlier ssh still held the Child's port. Now a flagged link that stays multi-route keeps its watch, with every process it already holds, re-keyed to the new instance. A flagged link with nothing watched is settled again from its pidfile. Both are observe-only from then on, which keeps the existing rule that no watch acts for a replaced record: no signal, and no pidfile removal, also when the processes end and when an unreadable file becomes readable later.

A running watch could end on an unreadable pidfile. The watch read its pidfile leniently. If the file turned unreadable and every watched process exited, the block lifted, although the file might now name another tunnel. The watch now reads strictly. In that case it hands the link to the per-tick re-read instead of lifting, and whatever the file names once readable is only observed, as for any replacement.

Removal and re-adding. Removing a link now clears its flag and re-read state with its watch. A link that a reload adds to this process is settled from its pidfile as at startup, so a link removed and added again with the same id is neither left blocked nor started past an earlier tunnel.

Comment. The READY_PROOF_MS comment in src/link/route-rounds.ts and the matching sentence in structure/remote-link.md now say that the first supervisor tick at or after a candidate's deadline stops it.

What keeps this inert. Everything here applies only to links with several routes, which exist only after an operator adds a route with ocx link routes add. For those links, behaviour changes only when a pidfile exists but cannot be read, when a flagged link's record changes or is removed and added again in a reload, or when a link is added by a reload (one extra pidfile read; a new link has no pidfile). One-route links take the old path, and no config key or default changes.

Verification

Tests written first, all in tests/clients/link-route-supervisor.test.ts. The harness gained a capture of the supervisor's warnings, and the pidfile cases use the supervisor's own reader on a mkdtemp directory. Every test below fails when src/link/supervisor.ts is put back to dev (36ee226); I ran the final test file against that file: 48 pass, 13 fail.

  • a multi-route link whose pidfile is a directory / holds no pidfile at start is blocked, warned once, and starts once the file is gone: on dev, orphan is undefined at start.
  • an unreadable pidfile that becomes readable is watched like any earlier tunnel until its process is gone.
  • a flagged one-route link that gains routes while its pidfile is a directory / holds no pidfile (the fix(link): cap a late grant at the first deadline and keep the orphan flag for an unreadable pidfile #8610 test, extended): on dev, deleting the file never lifts the block.
  • a reload that changes a flagged link's record keeps it blocked while the earlier tunnel may live (a listener-port change): on dev, the flag is gone after the reload.
  • a reload that changes the record of a link blocked by an unreadable pidfile keeps it blocked.
  • a reload that changes a watched link's record keeps watching the earlier process after its pidfile is gone.
  • after a reload that replaced its record, a watch only observes: no signal, and no pidfile removal when the process ends: on dev it hits the 5 s test timeout.
  • a pidfile unreadable at a reload that replaced the record is never acted on once it becomes readable.
  • a watch whose pidfile turns unreadable does not lift the block when its process ends.
  • a pidfile replaced while unreadable, after the watched process ended, is only observed once readable: on the previous commit of this branch it sent SIGTERM and SIGKILL to the replacement.
  • a blocked link that is removed and added again with the same id is settled from its pidfile, not left blocked.

The existing an orphan watch whose link was replaced sends no further signal and touches no pidfile still passes. It is what made the reload path observe-only.

Commands on the head (exit codes):

  • TMPDIR=/tmp bun test tests/clients/link-route-supervisor.test.ts tests/clients/link-supervisor.test.ts: 97 pass, 0 fail (exit 0).
  • bun run typecheck: exit 0.
  • bun run privacy:scan: exit 0.
  • bun run structure:check: exit 0.
  • TMPDIR=/tmp bun scripts/test.ts --parallel=2 tests/clients tests/server/link-management-routes.test.ts tests/test-layout.test.ts tests/test-layout-tooling.test.ts tests/ci-workflows/structure-ssot.test.ts tests/ci-workflows/ci-structure-gate.test.ts tests/ci-workflows/file-size-ratchet.test.ts (204 files, including tests/clients/link-boundary.test.ts): 3468 pass, 67 skip, 3 fail (exit 1). The 3 failures are the Rust ocx-child tests (tests/clients/link-ocx-child-e2e.test.ts, two cases, and the Rust case of tests/clients/link-ocx-child-conformance.test.ts). They stop before any test code runs, because cargo on the build machine has no default rustup toolchain. This branch touches no Rust code.

NOT RUN: the full bun run test suite (the build machine allows focused files only; CI covers the rest); the three Rust ocx-child tests above (no toolchain); the docs-site build (one sentence changed in the English guide, which has no translated copies); Windows and macOS (same code; without a readable start time a watched process is never signalled anyway, as before).

Review: four read-only rounds by one independent reviewer (gpt-6.1-terra, xhigh effort), continued across rounds.

  • Round 1 found two P1s and one P2: reload discarded the watch before reading the pidfile; the observe-only setting did not persist; the guide promised a start as soon as the file was readable.
  • Round 2 found a P1 and a P2: a watch ended on an unreadable pidfile; a removed link kept its flag.
  • Round 3 found a P1: a finished watch's handoff could signal a replacement process.

Each finding has a test in the list above. Round 4 found no P0–P2 and no regression, and ran the two supervisor test files on the head (97 pass).

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Implemented by Claude Opus 5.5 (anthropic/claude-opus-5-5) in GOAT through the Codex harness; in-lane review by GPT-6.1 Terra (xhigh). Received as a git bundle from the VM that built it and pushed by the coordinator.

… and on a changed record

At Home start the multi-route reaper read an unreadable pidfile as missing, so the
link's first round started while an earlier ssh might still hold the port. It now
uses the strict reader: an unreadable pidfile flags the link and warns once. A link
blocked that way is read again every tick, so making the file readable turns the
block into an ordinary watch and deleting it lifts the block without a restart.
A reload that changed a flagged link's record cleared the flag without reading the
pidfile; a new listener port changes every record and keeps every tunnel port. The
link is now settled again from its pidfile, and the new watch only observes.

Refs #8609
…its record, observe-only

The reload re-check deleted the existing watch before reading the pidfile, so a
pidfile that was deleted or replaced meanwhile lifted the block while the watched
ssh could still hold the port. The watch now stays, re-keyed to the new instance.
After a replacement it only observes: no signal and no pidfile removal, also when
its processes end and when an unreadable pidfile becomes readable later. The guide
no longer says a link starts as soon as its pidfile is readable again.

Refs #8609
…op a removed link's block

A watch read its pidfile leniently, so a file that turned unreadable while the
watched process exited lifted the block, although it might name another tunnel.
The watch now reads it strictly and hands an unreadable file to the per-tick
re-read, keeping the block. Removing a link clears its flag and re-read state, and
a link added by a reload is settled from its pidfile as at startup, so a re-added
id is neither left blocked nor started past an earlier tunnel.

Refs #8609
…hed watch handed it over

A watch whose processes all ended while its pidfile was unreadable handed the file
to the per-tick re-read with signalling allowed, so a replacement tunnel named by
the file once readable could be sent SIGTERM and SIGKILL. To that watch any later
file is a replacement, so the re-read now only observes it.

Refs #8609
@lidge-jun lidge-jun added the bug Something isn't working label Nov 13, 2026
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@lidge-jun
lidge-jun marked this pull request as ready for review November 13, 2026 01:48
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 13, 2026 01:48
@lidge-jun
lidge-jun merged commit f7c2131 into dev Nov 13, 2026
54 of 58 checks passed
@lidge-jun
lidge-jun deleted the fix/link-orphan-unreadable-pidfile-startup branch November 13, 2026 01:48
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T01:50:22.236487Z 9805aaf Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant