Skip to content

feat(link): let a Remote Link use several SSH routes, preferring direct ones over the relay - #8602

Merged
lidge-jun merged 23 commits into
devfrom
feat/link-routes
Nov 12, 2026
Merged

lidge-jun merged 23 commits into
devfrom
feat/link-routes

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

A Home-initiated Remote Link names one SSH alias, so when the only alias a Home uses for a Child is a relay alias, every model request of that Child crosses the relay even when a direct path (same LAN, a tailnet address) exists. Re-linking on a direct alias by hand works, but then the relay is gone as a fallback when the direct path is down.

This adds an ordered route list to a link and a selection loop in the Home's tunnel supervisor. Nothing in the product writes the list yet (the management API and ocx link routes commands come in a follow-up); a link without the field behaves exactly as before.

  • Record. LinkRecord.routes?: string[] (src/link/store.ts), 2 to 8 SSH aliases, direct first, relay last. It must contain alias, which stays the pinned host-key name and the destination of every exec operation. [alias] is normalized away, so a one-route link keeps the exact record an older Home reads. An older Home rejects a store that lists routes (strict RECORD_KEYS), the same downgrade caveat as remotePlatform and childKind.
  • One pin for every route. Tunnel, exec and the new route probe take an optional destination while HostKeyAlias stays the alias, with StrictHostKeyChecking=yes against the link known_hosts file. A route that offers another key fails with "Host key verification failed"; nothing is trusted on first use or written.
  • Selection (src/link/route-rounds.ts, src/link/route-select.ts). Routes are probed in order (10 s each, on the supervisor clock). The -R tunnel starts on the first that answers. OpenSSH's own debug1 line must confirm that the Child granted the remote listener; the stderr tap otherwise keeps exactly today's classifier lines. The tunnel is then proven end to end: the Child fetches its own /v1/catalog carrying a fresh proof id, its link relay forwards it with relay authentication, and the Home (src/link/route-proof.ts, hooked into the hub-link admission) accepts the id only for that link key, before its deadline, under the current tunnel generation, while the same request returns a catalog. Only then is the link connected (30 s per candidate). The readiness step reads the Child's ocx config show --json with the runner's 4 MiB cap, the same limit enrollment uses, so a Child with a large config ([Bug] Remote Link: Child preflight fails when ocx config show --json exceeds the 64 KiB SSH output cap #8356) works with several routes too.
    • Failed routes rest per class at today's per-alias rates: transient 1–30 s, forward 60 s, auth 5 min, host key 10 min.
    • The link reports the most recoverable outcome over all routes, so it fails terminally on the host key only when every route refuses the pinned key, for example a Child replaced by another machine.
    • A candidate without a proof at its deadline is stopped like a failed reconnect today. Requests on it fail toward the caller and are never replayed or duplicated. This differs on purpose from a one-route tunnel, which reads connected after its 5 s grace: a granted forward whose relay cannot authenticate looks the same as a slow one, and keeping it would starve the next route.
    • A proven tunnel is never stopped for a non-failure reason, and a link never returns to an earlier route by itself: moving a working link needs a drain protocol first. A direct route that was down when the link connected stays unused until the route in use fails, the Home restarts, or ocx link reconnect starts a new round from the first rested route.
  • Ownership. A link with several routes starts no tunnel while an earlier one may live. Its pidfile records the process start time. Only a process proven to be the pidfile's own (pid, argv, start time, re-read before every signal) is signalled. Unreadable or replaced identities are watched without signals until every watched process is gone. Status shows such a link as failed with reason stale tunnel may hold the port; stopping that ssh process (its pid is in the pidfile) lifts the block.
  • Status. GET /api/link/status?routes=1 adds routes, route ({name, index, since}) and routeBlocked to multi-route rows; ocx link status --json asks for it together with the link WebSocket counters (feat(link): the Home accepts the Responses WebSocket over a Remote Link, behind a closed release gate #8593) and the traffic counters (feat(link): measure the bytes each Remote Link carries and record request body size on log rows #8584), as ?transport=1&traffic=1&routes=1, and validates all of them strictly; a row can carry every block. The default document is unchanged, except that an ssh client that is not OpenSSH reads as a failed multi-route link with reason ssh_client_unsupported. Route names stay on the Home's admin/dashboard surface (not in the Child-readable relay list) and out of diagnostics, history and warnings.
  • SshRunner.run gains an opt-in AbortSignal with one termination path (SIGTERM, SIGKILL after 2 s, settle within 2 s more) for timeout and abort; callers without a signal are unchanged.
  • The hub-link admission takes the proof header off the request in one helper, takeRouteProof (src/link/route-proof.ts), before anything else handles it.
  • structure/remote-link.md has a new "Routes" section, including the readiness, return and orphan rules above.
  • The branch is merged with dev at 7e3cefd16 (feat(link): measure the bytes each Remote Link carries and record request body size on log rows #8584 traffic counters, Fix nested model selectors in Pi exports #8287, test(sync): expect Pi's server-owned refresh after #8287 #8606, feat(link): compress Child request bodies with zstd for a Home that accepts it #8594 zstd request bodies, feat(link): the Home accepts the Responses WebSocket over a Remote Link, behind a closed release gate #8593 link WebSocket on the Home). Conflicts in src/cli/link.ts, src/link/status-projection.ts, src/server/management/link-routes.ts and the link CLI tests were resolved by keeping both sides: the traffic block (?traffic=1) and the transport block (?transport=1) work as on dev, and the route fields as above. tests/cli/cli-link-activity.test.ts from dev was updated for the longer status query.

Unchanged by default: links without routes take the existing supervisor path (same argv, readiness, timings and orphan handling); the status document without ?routes=1; exec operations (sync, update, observe, revoke) still use the alias. structure/providers/codex-installation-identity.md and structure/usage-client-attribution.md own serve-options.ts and src/server/index/; they were reviewed and stay accurate.

Verification

Tests written to fail first, run against the base before the change:

  • tests/clients/link-routes-store.test.ts, link-routes-argv.test.ts and link-route-supervisor.test.ts failed to load on the base, because effectiveLinkRoutes and buildRouteProbeArgv did not exist.
  • tests/clients/link-ssh-runner-abort.test.ts failed 7 of 7: the runner had no abort contract.
  • tests/clients/link-route-select.test.ts and link-route-proof.test.ts cover new modules.
  • A row carrying the route fields, a traffic block and a transport block decodes with all three (tests/clients/link-routes-status.test.ts).
  • Final-review fixes: a supervisor case where the Child's config output is over 64 KiB failed on the old 64 KiB cap (the fake runner now enforces output caps like the real one) and connects now; takeRouteProof is tested on real requests (header removed and reported, absent, not removable).
  • Each review fix added a supervisor case that failed on the previous commit and passes now. These cover: an untracked candidate after a failed pidfile write; unreadable orphan identity; work started during a reload; a late proof; a late probe; a replaced pidfile; pid reuse.

Commands, with exit codes. The full set ran on 53c2f1209; the head adds only a structure/remote-link.md wording fix (01f954df7), after which structure:check, privacy:scan and the structure, layout and repo-hygiene tests were run again (0, 0, 69 pass).

TMPDIR=/tmp was used because the build machine's default temp path makes local-control-socket.test.ts's Unix socket path too long.

NOT RUN: the full bun run test suite (the build machine allows focused files only; CI covers the rest); the three Rust ocx-child tests (tests/clients/link-ocx-child-e2e.test.ts and the Rust case of tests/clients/link-ocx-child-conformance.test.ts), which have no toolchain there; Windows and macOS behaviour, which uses the same argv builders (Windows Child proof via the existing .NET loopback request); the real-OpenSSH fixture in CI (opt-in).

Design and code were reviewed by an independent reviewer: 14 plan rounds, then 7 code rounds, ending with no open P0–P2; the #8584 merge had a fresh final review with no findings. A fresh final review of 00c8abfa3 found one P2 (the 64 KiB config cap) and seven P3s; all are answered in this head: fixed, documented (return rule, readiness deadline, orphan recovery) or, for the unbounded wait in stopChild, kept as on dev with the reason in the structure page. The author's own review of those fixes and of the #8593 merge had not finished at handover.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed. structure/remote-link.md is updated; the user guide comes with the management commands in the stacked PR, since nothing user-facing writes routes yet.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. Covered: host-key pinning on every route; no trust on first use; route names refused as options and kept out of logs; the proof id is single-purpose, bound to key and tunnel generation, and removed before dispatch; the Child relay still strips caller credentials.

Implemented by Claude Opus 5.5 (anthropic/claude-opus-5-5) in GOAT through the Codex harness. Received as a git bundle from the VM that built it and pushed by the coordinator.

…a confirmed-termination ssh run

A link record may list several SSH routes to the same machine. The alias stays the pinned host-key name and is required in the list; a one-route link keeps the exact record an older Home reads. Tunnel and exec argv can connect to a route while HostKeyAlias stays the alias, and buildRouteProbeArgv checks a route strictly against the pinned key. SshRunner.run gains an opt-in AbortSignal with one termination path for timeout and abort.
…rrelated readiness proof

A link with several routes is connected by selection rounds: each eligible route is probed against the pinned host key, the -R tunnel starts on the first that answers, OpenSSH must confirm the granted remote listener, and the Child must fetch its own catalog with a fresh proof id that this Home admits under the tunnel generation. Failed routes cool per class at today's per-alias rates; the link reports the most recoverable outcome over all routes. One-route links keep the existing path.
…ssion

A key-admitted catalog request on the hub-link listener reports the proof id it carried; the optional listener set owns the proof registry and checks it against the tunnel generation the supervisor reports.
GET /api/link/status?routes=1 adds routes, route (name, index, since) and routeBlocked to links with several routes; the default document is unchanged. ocx link status asks for it and validates the fields strictly. An unsupported ssh client reads as a failed multi-route link.
…enSSH

ssh -G shows the pinned alias, strict checking and the link known_hosts win over a hostile per-destination config. An opt-in fixture (OCX_OPENSSH_ROUTE_FIXTURE=1) runs a private sshd -i through ProxyCommand: the pinned probe passes, another key is refused without touching known_hosts, a granted -R carries bytes, and a second session or a server without forwarding is refused with OpenSSH's failure line.
…, and take the proof header off the request

A tunnel is owned and watched before its pidfile is written, and a failed write stops it before the next route. An orphan whose identity cannot be read keeps the link blocked and is never signalled; a link whose orphan is gone starts its first round. New probes and tunnels wait while lifecycle work runs, and changed links lose their round before the first stop is awaited. A proof that settles after its candidate exited cannot promote it. The hub-link catalog admission deletes the proof header before dispatch.
…thout a deadline, and recheck proof deadlines

A multi-route link blocked by an earlier tunnel keeps a non-signalling watch and starts its first round once the process is gone or its pid names another process. A probe that settles while lifecycle work runs moves to a deferred phase no deadline applies to. A readiness proof promotes only before both its attempt and candidate deadlines.
…te probe results

An orphan watch belongs to one multi-route record instance and pidfile; removal, replacement or a change to one route retires it before any further signal, and a pidfile is removed only when it is still exactly the watched one. A route probe that settles after its deadline counts as a timeout even before the tick runs.
…te probe answer as a timeout

An orphan watch signals, lifts the block or starts a round only while the pidfile is still exactly the one it watches; a replacement is only watched and keeps the link blocked. A route probe that answers after its deadline is a transient timeout whatever its exit code or stderr says.
…laced

An orphan watch keeps every process its pidfile has named; after a replacement nothing is signalled, and the link stays blocked until all of them are gone.
@lidge-jun lidge-jun added the enhancement New feature or request label Nov 12, 2026
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

# Conflicts:
#	src/cli/link.ts
#	src/link/status-projection.ts
#	tests/cli/cli-link-check.test.ts
#	tests/cli/cli-link.test.ts
#	tests/server/local-control-socket.test.ts
# Conflicts:
#	src/cli/link.ts
#	src/link/status-projection.ts
#	src/server/management/link-routes.ts
#	tests/cli/cli-link-check.test.ts
#	tests/cli/cli-link.test.ts
#	tests/server/local-control-socket.test.ts
@lidge-jun
lidge-jun marked this pull request as ready for review November 12, 2026 23:34
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner November 12, 2026 23:34
@lidge-jun
lidge-jun merged commit 44c3d7c into dev Nov 12, 2026
60 of 66 checks passed
@lidge-jun
lidge-jun deleted the feat/link-routes branch November 12, 2026 23:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T23:38:21.101969Z 01f954d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

lidge-jun added a commit that referenced this pull request Nov 12, 2026
…anage

The branch already held #8602's head 01f954d, whose tree equals the squash's, so the tree is unchanged; this commit only records the squash as an ancestor.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01f954df79

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/link/store.ts
Comment on lines +34 to +38
* Ordered SSH routes to the same machine, direct routes first and the relay alias last.
* Absent means exactly [alias]; a list always contains alias, which stays the pinned host-key
* name (HostKeyAlias) and the destination of every exec operation. Hub-initiated links only.
*/
routes?: string[];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document multi-route configuration in the public Remote Link guide

The new routes field is user-facing configuration, and the structure document says that manually editing the record is currently the only way to enable it, but this commit makes no corresponding docs-site/ update. Users therefore lack public instructions for configuring the feature, including the required alias entry, the lack of automatic failback, and the fact that older releases reject the modified store on downgrade. Add these instructions and compatibility warnings to the public Remote Link guide, as required by src/AGENTS.md.

AGENTS.md reference: src/AGENTS.md:L29-L29

Useful? React with 👍 / 👎.

Comment thread src/link/supervisor.ts
Comment on lines +711 to +712
if (routesEdited.some(record => rounds.isMultiRoute(record))) rounds.recheckClient();
for (const record of routesEdited) if (rounds.isMultiRoute(record)) rounds.routesChanged(record);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconsider terminal host-key failure when adding a new route

If every existing route has failed host-key verification, adding a new, working route and reloading never tries it. This branch updates the route books but preserves the reducer's terminal failed(hostkey) state, which has no retryAt; consequently, spawnFor rejects the reload attempt and every subsequent tick, even though the newly added route has never refused the pinned key. A focused reproduction remained terminal after adding a working direct route and advancing the clock by an hour. Recompute or reset the terminal state when a route-list edit introduces an untried route, while preserving any existing live tunnel.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant