Repository navigation
fix(link): operation journal follow-ups: a link kept down (and shown) after an uncertain revoke, durable revoke marks, and a journal that cannot lock a Home out - #8638
Merged
Conversation
… changed POST /api/link/operations/<unknown>/resolve (or ocx link operations resolve with a typo) on a Home that never journalled answered 404 but created an empty operations.json, and from then on every plain apply and revoke took the alias lock and read the journal. updateOperationsJournal now writes only when the change or the prune altered the journal.
A failed remote_cleanup intent ran after stopLink, so a listed link kept its tunnel stopped and the caller got an unmapped 500 without the operation id. A failed remote_cleanup completion after a good purge reported remote_disconnect_failed and restarted the tunnel toward a purged Child, and a failed record_removed completion skipped closing the link's sockets. The intent is now written before stopLink; strict intents answer 500 operation_journal_failed with the operation id before their effect; completions are best effort, and a purge receipt whose mark failed is written with the outcome. linkOperationMarkHook lets tests fail one chosen mark.
…play The replay branch asserted that the entry it had just read was still there. If another journal write pruned it in between, remove() returned null and the DELETE was left unhandled. It now falls through to the normal revoke once the replay has released the alias lock.
…in the CLI's 409 An invalid DELETE body on a link a Child job holds answered 400 since revokes were journalled; before, it answered 409. The body is judged after the Child-operation reservation again (404, then 409, then 400). Plain ocx link add called the in-process 409 "another enrollment or revoke", which a concurrent probe, confirmation or route edit also returns; it now says another link operation for this alias.
…it with the client The wait polled for up to 360 s, the CLI's whole enrollment deadline, and ignored the client: an apply could start after the CLI had printed that its outcome was uncertain, and a waiting revoke held its link's Child-operation reservation the whole time. The wait is now at most 120 s and ends at the next poll once the request's signal aborts, changing nothing.
…or the alias Recovery settled unknown and left the reserved key in place when another link held the alias, although the reserved key id names only this operation's key. It now revokes that key when no record carries the reserved link id, and still settles unknown.
exclusive changes no tunnel argv, yet being part of the record instance made a hand edit from exclusive to a route list an instance change, whose reload path drops the orphan watch without the pidfile check, and restarted the tunnel for nothing. The instance keeps explicitRoute, which does change the argv.
The journal had no size limit, and every mark rewrites the whole file under the config mutation lock. A new operation past 256 entries (after pruning) now answers 409 operation_journal_full before any effect; marks on existing entries are always written and residual or unknown entries are never pruned to make room. 256 is about ten times what ten disposable machines need over the 30-day retention.
- ocx link revoke --operation-id exits 4 on link_not_found: nothing was
journalled, so it no longer prints {linkId} with exit 0.
- The server refuses operationId without fingerprint, as the CLI does.
- A committed enrollment whose plant completed answers planted: true, live and
on replay; the CLI prints it on a replay too.
- The alias lock refuses a directory inside the real OpenCodex home under test,
before any mkdir, like the journal writer (its test was written after the
guard: before it, the call would have written the live home).
- A damaged journal's 500 names the way out, and a plain --force revoke still
runs without the open-entry check.
…st intent before stopLink Review round 1: a purge receipt whose completion mark failed was held only in memory while the key and the record went, so a crash before the outcome was written lost the Child's purged:true. That completion is now strict: the revoke stops before the key with 500 operation_journal_failed (unknown) and leaves the tunnel down toward the purged Child. A journalled force revoke wrote its first intent (key_revoked) after stopLink, so a failed write left the tunnel stopped and recorded failed_clean; the first applicable intent is now written before stopLink.
…Child is unsettled Review round 2: when a purge receipt could not be written, the revoke stopped unknown with the link still recorded, but the supervisor hold was released, so any later reload or a Home restart reconnected the tunnel to a purged Child. The supervisor now skips spawning a tunnel for a link whose revoke wrote its remote_cleanup intent and then settled unknown, or is still open with no revoke in this process running it (the process died). A cleanup that failed and that the revoke answered by restarting the tunnel (remote_disconnect_failed, purge_unsupported) suspends nothing, and a running revoke is registered as live until its outcome is written so its own restart goes through. Resolving the entry reloads the supervisor and lifts the suspension. A Home that never journalled pays one stat per spawn attempt; a damaged journal suspends nothing.
…eak a revoke's live mark Review round 3: remote_disconnect_failed covers a timeout and a purge with no report, after which the Child may already be disconnected or purged, yet the revoke restarted the tunnel and the suspension exempted that code. And a revoke that threw before finish kept its live registration, so the supervisor treated the abandoned entry as running. A journalled revoke now restarts the tunnel only when the Child's disconnect provably never ran (platform discovery failed, the purge's runtime inspection failed or found a full install, no planted binary to run) and settles failed_clean; any other cleanup failure leaves the tunnel down and settles unknown, and every unknown revoke past its cleanup intent suspends its link until resolved. A plain revoke restarts the tunnel as before. The live mark is released in a finally around the whole revoke.
Review round 4: removeReserved released the supervisor hold in its finally while the entry was still open and marked live, so a reload landing before finish wrote the unknown outcome could spawn a tunnel toward a Child that may be purged. A revoke that leaves the tunnel down now drops its live mark in the same synchronous finally, just before the hold, so that reload finds the link suspended; only a revoke that restarted the tunnel itself after a cleanup that never ran stays live until finish.
Contributor
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Contributor
|
✅ Deterministic PR hygiene checks passed. |
…ended and status reads revoke_unsettled Final review of #8638: after a journalled revoke settled unknown past its remote_cleanup intent (or died open), ocx link reconnect and the dashboard's Retry stopped nothing, spawnFor declined the link silently, and the route still answered 200 {linkId}; status showed the link idle since its creation with no reason. The supervisor's reconnect now answers "suspended" before it stops anything (and again just before the spawn), and its status carries the revoke that keeps the link down. The route answers 409 link_suspended with the operation id and state and the way out: resolve an unknown revoke, or repeat an open one with the same flags first. The status row reads state failed, since the revoke settled, reason revoke_unsettled; its keys are unchanged, so every CLI's strict status check accepts it. The CLI prints its own next step from the validated operation id (never the server's text) and exits 5.
…re pruned oldest first Final review of #8638: at 256 entries every new journalled operation was refused with "resolve residual or unknown operations", but resolving only starts the 30-day retention and committed or failed_clean entries cannot be resolved at all, so a provisioning loop that retried with a fresh id after each failed_clean locked the Home out of journalled operations for up to 30 days. Only unsettled entries (open, and unresolved residual or unknown) now count: at 256 of them a new operation is refused and the message names the way out. Settled entries never block; past 256 the oldest (by when they finished or were resolved) are pruned. The file holds at most 512 entries, about 0.5 MiB.
One conflict, the import block of src/link/supervisor.ts: dev's "CANDIDATE_KILL_GRACE_MS, createRouteRounds, type LinkRouteStatus" import is kept and this branch's operations-journal import is added after it. The rest of both sides merged on their own: a suspended link whose child is draining is still escalated and forgotten when it exits, and spawnFor then declines it; a revoke's hold-then-stopLink now ends within about 2 s for an ssh that ignores SIGTERM.
…ed revoke Final review of #8638: the guide covered only the unknown case, and nothing said how to leave a suspension when the Home stopped during the revoke (resolve refuses an open entry). The guide (English and Spanish), the capability registry with its generated skill surface, and structure/remote-link.md now say that a suspended link reads failed with reason revoke_unsettled, that reconnect answers link_suspended (exit 5), that resolving a revoke reloads the tunnels, and that an open revoke is repeated with the same operation id and --purge flag before it is resolved. ocx link reconnect no longer claims to always emit {linkId}, and ocx link operations names the bounds of the journal.
…wins over an orphan; one journal read per status Review round 6: pruning settled entries past 256 could drop one that finished a moment ago, so a caller retrying a committed enrollment whose answer was lost got link_exists (or started a new enrollment) instead of its recorded outcome. A settled entry is now pruned early only once it is a day old; a new operation is refused only after 1024 operations settled within a day, with a message to retry once the oldest is a day old, so a runaway loop is held for at most a day. A suspended link with an unverified orphan now reads revoke_unsettled, the step to take. The supervisor's status reads the journal once per snapshot instead of once per link.
…lias-lock wait Review round 7: an entry became prunable the moment it was a day old, so a retry that started just before that and then waited on the alias lock (up to 120 s) could find its entry gone. The early-pruning floor is now a day plus the full alias-lock wait and a minute (OPERATION_PRUNE_FLOOR_MS), and the burst refusal counts entries younger than that same floor.
Review round 8 (P3): the burst refusal said to retry once the oldest entry is a day old, but it counts entries until they are a day and three minutes old (OPERATION_PRUNE_FLOOR_MS). The response, the capability text and the structure doc now name the full floor.
No conflicts: dev's change to src/server/management/link-enrollment.ts and structure/remote-link.md merged with this branch's edits on its own.
lidge-jun
marked this pull request as ready for review
November 13, 2026 05:16
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
lidge-jun
added a commit
that referenced
this pull request
Nov 13, 2026
… intent, as #8638 orders the revoke #8638 moved a force revoke's key_revoked intent ahead of stopLink, so the merge left the tombstone written after that intent and after the tunnel stopped. The tombstone is now written first, before any intent and with the tunnel untouched (a refused write needs no restart), and it is dropped when the first intent or the record_removed intent cannot be written, as it already was when the key revoke or the record removal fails. Tests: a damaged tombstone file refuses before the tunnel is touched; a journal that breaks during the revoke answers 500 operation_journal_failed, keeps the link and drops the tombstone.
5 tasks done
lidge-jun
added a commit
that referenced
this pull request
Nov 13, 2026
Brings in dev through de85cf2, including #8638 (operation journal follow-ups: strict revoke intents, a link suspended after an uncertain revoke, journal size bounds, write-only-on-change) and #8646. Conflict resolution keeps both sides: - operations-journal: dev's bounded pruning and write-only-on-change, with this PR's keep pin honoured by both the retention filter and the over-limit prune. - link-operation: dev's room check, live marks, strict intents and cleanupNotRun/release on RevokeJournal, with this PR's pinned writes, operationId and beforeReceiptCleared; the resumed retirement journal gets no-op cleanupNotRun/release (its link is held for the resume). - link-routes remove(): the early replay branch resumes the operation's own receipt before replaying; dev's 404/409/400 order and plain-force open-entry exemption stay, and a plain force still settles and disowns an open owner's receipt. - link-routes removeReserved(): this PR's receipt flow (listed check, cleanup evidence, mode check, claim before the locked commit) with dev's first intent before stopLink, strict key/record intents (the record intent inside the locked step, before the store write), the cleanup-failure rule and the live-mark release.
lidge-jun
added a commit
that referenced
this pull request
Nov 13, 2026
…cx-child owner record /2, detach --expect-link, ocx link recover) (#8612) * feat(ocx-child): name the link in the CODEX_HOME owner record and add detach --expect-link Attach now writes ocx.child.codex-home-owner/2 with the link id while it holds the lifecycle and CODEX_HOME locks, in the attaching phase, before any CODEX_HOME write. detach --expect-link <id> restores only when state.json and the owner record both name that link (or nothing is left at all); a /1 record or a missing state answers residual (exit 6), another link, an unreadable record or another OPENCODEX_HOME answers conflict (exit 5), and neither changes or purges anything. * feat(link): keep a tombstone for force-revoked ephemeral links and add POST /api/link/recover A forced revoke of a Home-initiated ephemeral link now writes <configDir>/link/tombstones.json (link id, alias, pinned fingerprint, platform, routes) before the key and record go, and refuses the revoke if it cannot. POST /api/link/recover {linkId} runs the managed planted ocx-child's disconnect --purge --json --expect-link <id> over the alias with the pinned host key and removes the tombstone only on an exit 0 report naming that link with purged:true. {operationId} resolves through a slot the O1 enrollment journal registers; until then it answers operation_journal_unavailable. * feat(cli): ocx link recover, its capability, docs and structure ocx link recover (--link-id <id> | --operation-id <id>) posts to POST /api/link/recover and accepts only a receipt naming that link with purged:true. A forced revoke of an ephemeral link now names it in its recovery instruction. structure/remote-link.md documents the /2 owner record, detach --expect-link and the tombstone; the guide and the CLI lifecycle reference list the verb. * fix(link): recover never restores another link, settles a lost receipt, purges under the lock Review round 1 on #8612: - detach --expect-link also reads the link an unresolved attach rollback would put back; an interrupted attach between two links answers residual for those links (conflict for others), so recovery never restores and detaches a link it was not asked for. - --purge removes the planted binary before the lifecycle lock is released, so a waiting attach cannot start before the binary is gone. - A recover whose receipt was lost after the purge settles through a read-only check that neither the managed binary nor any ocx-child recovery record remains. - Recover failures answer fixed hints, never the machine's stderr. * fix(link): settle a lost recover answer only on the Child's link-bound receipt Review round 2 on #8612: - detach --expect-link writes $OPENCODEX_HOME/link-receipts/<id>.json (ocx.child.detach-receipt/1) before its purge and again after it; recover reads that receipt when the managed binary is already gone and settles only on one naming the link with restored and purged true. The absence-of-files check is gone: absence proves nothing. - The rollback snapshot is parsed and home-checked as recovery will before anything changes, so a malformed one is a conflict with no phase change. * fix(ocx-child): guard the shared planted binary and bind restore receipts to one attachment Review round 3 on #8612: - Every relay serving from the managed planted binary holds .ocx-child-in-use.lock beside it shared; a purge takes it exclusively without waiting and keeps the binary (purged:false) while a relay of any home runs from it, and serve refuses while a purge holds it. - The restore receipt is written once, before the purge, and records only the restore; recover reads the purge from the missing binary, so a crash after the purge stays provable. - An attach of a link removes that link's old receipt under the lifecycle lock first. * fix(link): keep the in-use lock file, honour a kept binary on rollback, prove the purge by absence Review round 4 on #8612: - The in-use lock file is never unlinked, so no relay can lock a fresh file of the same name while a purge removes the binary. - The enrollment rollback purge leaves a binary the Child kept for another home's relay (POSIX: purged:false in detach --purge --json; Windows: still present after the purge). - The receipt read exits non-zero while a planted file of that name exists, executable or not, or its directory cannot be searched, so only a confirmed absence proves the purge. * fix(link): the receipt read trusts an absent binary only below searchable directories Review round 5 on #8612: a test for a path reads false both when it is absent and when an ancestor cannot be searched. The POSIX receipt read now walks from $HOME down to the planted binary and exits 4 on any directory it cannot search; the Windows read counts only Get-Item's ItemNotFoundException as absent. * feat(cli): advertise link.recover/1 in ocx capabilities --contracts GOAT reads the Home's contracts to know it can recover a force-revoked ephemeral Child. * test(link): recover's platform probe and receipt read go over the explicit route too * fix(link): pass the link id to the Windows recover scripts as data, never spliced into source * fix(link): bound tombstones with list and forget, settle them on uncommitted or ordinary revokes, fold alias case, name a kept in-use binary, refuse an unreadable prior snapshot * fix(link): forget a tombstone only from the CLI on the Home while its link is gone, and clean up only the tombstone a failed revoke wrote * fix(link): take the alias lock for apply and revoke whenever a tombstone names the alias, so recover excludes them * docs(link): mirror ocx link recover into the Spanish guide and lifecycle reference; settle a recover whose binary stays for another home; drop the tombstone when the revoke journal fails * fix(link): write a forced revoke's tombstone before its first journal intent, as #8638 orders the revoke #8638 moved a force revoke's key_revoked intent ahead of stopLink, so the merge left the tombstone written after that intent and after the tunnel stopped. The tombstone is now written first, before any intent and with the tunnel untouched (a refused write needs no restart), and it is dropped when the first intent or the record_removed intent cannot be written, as it already was when the key revoke or the record removal fails. Tests: a damaged tombstone file refuses before the tunnel is touched; a journal that breaks during the revoke answers 500 operation_journal_failed, keeps the link and drops the tombstone.
3 tasks done
lidge-jun
added a commit
that referenced
this pull request
Nov 13, 2026
…rom a truth table; the operation journal keeps replays for at least 2 hours (#8654) * fix(link): the CLI explains a full operation journal and a journalled revoke that did not complete Final review of #8638: linkRequest replaces every API error with "Link management request failed (N)", operation_journal_full was not among the add refusals, and runRevoke rethrew everything but link_not_found, so a full journal, or a repeated open revoke that recovery settled unknown (409 operation_interrupted), printed only that generic line. operation_journal_full now carries a closed bound field (unsettled or recent); it is additive, and an older CLI drops it and still refuses. ocx link add and ocx link revoke with --operation-id print their own guidance for it, with the code and the bound, keeping exit 5. A journalled revoke that did not complete prints "(operation <id>: <outcome>) (code <code>)" from the filtered body (the caller's validated id, a closed outcome, the filtered code) and names ocx link operations resolve when the outcome is unknown. No server text is printed. * docs(link): say what an open revoke means and that its flags cannot be read back Final review of #8638: the open-suspension text said "the Home stopped during it", but an open entry also means the revoke failed before writing its outcome; "add --purge if it was a purge" cannot be checked from the entry, which records only a digest of the flags, so a wrong guess answers operation_mismatch; and the ALIAS_LOCK_WAIT_MS comment promised a waiter time to finish before the CLI deadline, which an apply's worst case does not give. The route, the CLI, the capability text, the guide (English and Spanish) and structure/remote-link.md now say "did not record its outcome (the Home stopped or the revoke failed before writing it)" and tell the caller to repeat with the flags of the first attempt, naming operation_mismatch for a different --purge. The alias-lock comment says the bound limits the wait and promises no time to finish. * docs(link): state the operation journal's throughput limit by churn Final review of #8638: the comment on OPERATION_JOURNAL_MAX_RECENT_SETTLED argued by machine count. The limit it sets is a burst of 1024 and about 42 journalled operations per hour sustained (1024 / 24.05 h); ten disposable machines that live ten minutes each make about 120 per hour and reach it in about 8.5 hours. The comment, structure/remote-link.md and the ocx link operations capability text now say so. The constants are unchanged until the owner chooses a replay horizon or a higher cap. * docs(link): a suspension needs the revoke's cleanup intent and no live revoke Lane review (P3): the ocx link operations resolve text said any revoke that did not record its outcome keeps the link down, but suspension needs the remote_cleanup intent, and an open entry only when no revoke in this Home still runs it. The capability text and the generated skill reference now say so. * docs(link): the cleanup-intent condition covers an open revoke too Final review of #8654 (P3): the ocx link operations resolve text tied the remote_cleanup intent to the unknown case only, but suspendingRevoke requires it for an open entry as well, so an open forced revoke or an open revoke of a Child-initiated link never suspends. The capability text and the generated skill reference now read as structure/remote-link.md does. * fix(link): a journalled revoke refused before it ran says nothing was changed Final review of #8654 (P3): operation_mismatch, link_operation_open, link_operation_in_progress and operation_journal_unreadable on ocx link revoke --operation-id printed 'Link revoke did not complete (code X)'. They now get the CLI's own text that nothing was changed, with the code, as add's refusals do. The list is closed: any other code, including ones a newer Home adds, keeps the generic text and makes no such claim. Exit codes are unchanged. * fix(link): the journalled revoke helper re-checks the outcome and code itself Final review of #8654 (P3): journalledRevokeError accepted any string outcome and was safe only because linkRequest's filter had limited it. It now re-checks the outcome against OPERATION_OUTCOMES and the code against the filter's pattern, as journalFullGuidance re-checks bound; a test passes an unfiltered body straight to the helper. * docs(link): say when the CLI adds the operation note and the bound Final review of #8654 (P3): structure/remote-link.md said an unfinished journalled revoke always prints the operation note and a full journal always the bound. The operation part appears when the answer names this operation and a known outcome, the bound when the Home sends it; the doc now says so and lists the refusals that say nothing was changed. * docs(link): a cleanup intent means cleanup may have started; a full journal also says nothing changed Lane review (P3s): the remote_cleanup intent is written before the tunnel stops and before the Child is contacted, so the capability text now says the Child's cleanup may have started instead of that it reached the Child (generated skill reference regenerated). structure/remote-link.md lists operation_journal_full's guidance among the texts that say nothing was changed. * fix(link): a forced revoke refused at the tombstone limit says nothing was changed After #8612, ocx link revoke --force --operation-id of an ephemeral Child can be refused with tombstone_limit (409) or tombstone_write_failed (503). tombstone_limit is answered before the tombstone is written, before the first journal intent and before the tunnel stops, so it joins the CLI's closed list of refusals that say nothing was changed; tests/server/link-recover.test.ts now pins that the key is not revoked and the tunnel not stopped. tombstone_write_failed stays generic: the tombstone file may already be written when its permissions step fails. * feat(link): keep a settled journal entry replayable for 2 hours instead of a day Decision for disposable-machine churn: with a one-day replay horizon the 1024 recently-settled bound allowed about 42 journalled operations per hour, and ten machines that live ten minutes each (about 120 per hour) hit it after about 8.5 hours. OPERATION_REPLAY_MIN_MS is now 2 hours; the alias-lock wait and the minute of margin stay, so the prune floor is 2 h + 120 s + 60 s, and the 1024 bound stays: about 500 journalled operations per hour sustained, 4.2 times the example churn, with a worst-case file of about 1.5 MiB. The guarantee: a repeat with the same operation id within 2 hours of the answer always gets that answer; later it may be treated as a new operation, which meets a safe refusal (link_exists for an enrollment whose link is there, link_not_found for a revoke whose link is gone). Unsettled entries are never pruned and the 30-day retention is unchanged, so the window matters only after more than 256 settled entries. The constant's comments, the 409 text for the recent bound, the CLI's own guidance, the ocx link operations capability text (and its generated skill reference) and structure/remote-link.md say so. * docs(link): state the replay guarantee with its conditions, and pin a repeat after the window Lane review of the 2-hour horizon: 'a repeat within 2 hours always gets that answer' overstated it. A repeat that reaches the journal within 2 hours gets the recorded outcome (status, code and outcome, not the identical body); a refusal before the journal is read (the alias lock busy past its wait, an unreadable journal) changes nothing and is retried. After the window the id may be treated as a new operation, and that is not always a refusal: an enrollment whose link was removed meanwhile enrolls the machine again, now pinned by a regression test. The constant's comment, the capability text (and its generated reference) and structure/remote-link.md say so. * test(link): pin a repeat after the window through real pruning and a revoke; the replay needs the same parameters Lane review (P3s): the post-window test now revokes the link through the route, lets 256 later operations settle, advances the clock past the prune floor and asserts that a journal write pruned the original entry before the repeat enrolls again. The replay guarantee in the constant's comment, the capability text (and its generated reference) and structure/remote-link.md now says the repeat must use the same parameters; another set answers operation_mismatch. * fix(link): a tombstone_limit refusal says a repeat of its operation id replays it Final review of #8654 (P3): a forced revoke refused at the tombstone limit is recorded failed_clean, so repeating the same operation id returns the same refusal even after the person recovered tombstones. The CLI's guidance now says so and to retry with a new operation id. Its test now uses --force, the only way to meet that limit. * docs(link): the guide's replay sentence names its conditions Final review of #8654 (P3): the Remote Link guide (English and Spanish) said a repeat of the same command returns the recorded outcome without conditions. It now says: with the same flags, within 2 hours of its answer, as the structure doc and capability text do. * test(link): the replay test's comment names the replay window, not a day Final review of #8654 (P3): a comment in tests/server/link-enrollment-operation.test.ts still said the burst settles 'within the same day'. It now says inside the replay window. (The other half of this item, a tombstone_limit test sent for a --purge revoke, is fixed in 825132f, which moved it to --force.) * fix(link): link_operation_in_progress claims only that the link, key and Child were untouched Found while re-reading the merge of #8416: a resumed revoke (one that owns its retirement receipt) can re-own that receipt and then answer 409 link_operation_in_progress when a Child job holds the link. The link, its key and the Child are untouched, but 'Nothing was changed' is too strong for that path. The CLI text for this code now says the attempt did not touch the link, its key or the Child; the other refusals keep 'Nothing was changed'. structure/remote-link.md says so. * fix(link): tombstone_limit guidance holds for a resumed revoke too Lane review (P2): a resumed force revoke can re-own its retirement receipt before removeReserved meets the tombstone limit (#8416), and such an operation can stay open or unknown, so 'Nothing was changed', 'recorded as failed_clean' and 'retry with a new operation id' could all be wrong for it. The CLI now says the attempt did not touch the link, its key or the Child, and to check ocx link operations --operation-id <id> before reusing the id: a failed_clean entry replays this refusal (use a new id), any other state is repeated with the same id. structure/remote-link.md says so. * docs(link): the guide's replay sentence says the repeat must reach the journal Lane review (P3): the guide (English and Spanish) said a repeat within 2 hours returns the recorded outcome; a lock-busy or unreadable-journal refusal can stop it first. It now says a repeat that reaches the journal within 2 hours returns it, and to retry a transient refusal such as link_operation_in_progress, as the capability and structure text do. * fix(link): tombstone_limit guidance repeats the same operation id first Lane review (P2): failed_clean alone does not mean a fresh refusal. A failed_clean operation can still own a retirement receipt; if its resumed force revoke meets the tombstone limit, the entry stays failed_clean, and once capacity is freed the original id resumes and finishes while a new id is refused with link_retirement_owned. The CLI now says to repeat the same operation id first and to use a new id only if that repeat answers the same refusal again. tests/server/link-revoke-operation.test.ts pins the server side (refused at the limit, a new id refused with link_retirement_owned, the same id committed after a slot is freed); structure/remote-link.md says so. * fix(link): tombstone_limit guidance never declares a refusal recorded Lane review (P2): a second tombstone_limit does not prove the refusal was replayed; a resumed revoke meets the live limit again while capacity stays full, and its receipt still belongs to the original id. The CLI now says to free capacity and repeat the same operation id, to try a new id only if the refusal comes back although capacity is free, and to keep the original id if the new one answers link_retirement_owned (a safe refusal). The server test now repeats the resumed revoke with capacity still full (tombstone_limit again) before a new id is refused and the original id commits once a slot is freed. * fix(link): tombstone_limit no longer says the link and its key stay active Final review of #8654 (P2): a resumed forced revoke can meet the tombstone limit after an earlier attempt of the same operation already revoked the key and stopped the tunnel, then failed before the record went and dropped its own tombstone; the repeat re-records one and is refused. 'so the link and its key stay active' was then false. The text keeps only 'this attempt did not touch the link, its key or the Child', which holds on every path, also once #8661 drops the tombstone on every exit before the commit. * fix(link): tombstone_limit guidance names the owner of an unfinished removal instead of claiming it Final review of #8654 (P2): 'if that answers link_retirement_owned, the removal still belongs to this id' can be false. A fresh refusal recorded failed_clean owns no receipt, and another removal of the same link (a journalled operation or a plain revoke) may hold it; repeating this id then replays the same refusal and loops. The text now says an unfinished removal holds the link, and to repeat the operation id that owns it (ocx link operations lists the link's revokes) or finish it with ocx link revoke --force without --operation-id. * fix(link): closed guidance for an interrupted removal on a journalled revoke Final review of #8654 (P3): link_retirement_owned and link_retirement_mode_mismatch (#8416) printed only 'Link revoke did not complete (code X)'; the owning operation and the way out were in the server's message, which the CLI never prints, although the tombstone_limit guidance points the person at link_retirement_owned. They now get closed texts that name where to look (ocx link operations, a plain revoke without --operation-id, the same options, --force) and claim nothing about what an earlier attempt changed. structure/remote-link.md lists them. * docs(link): the replay guarantee names a revoke that resumes its own removal Final review of #8654 (P3): a revoke repeat whose own retirement receipt is outstanding resumes that removal first and answers that attempt (#8416), not the recorded outcome; the PR's own test repeats an entry recorded failed_clean and gets tombstone_limit, then committed. structure/remote-link.md, the capability text (and its generated reference), the constant's comment and the guide (English and Spanish) now say so. * docs(link): keep the guide's outcome list next to 'recorded outcome' Final review of #8654 (P3): '(retry a transient refusal such as link_operation_in_progress) (committed, failed_clean, residual or unknown)' read as if the second parenthesis listed transient refusals. In the English and Spanish guide the outcome list now follows 'recorded outcome' and the retry advice is its own clause. * test(link): point the tombstone_limit comment at journalledRevokeError Final review of #8654 (P3): the comment in tests/server/link-recover.test.ts said the CLI prints 'nothing was changed' for tombstone_limit through REVOKE_REFUSALS; at the head it comes from journalledRevokeError and says only that the attempt did not touch the link, its key or the Child. * docs(link): the journal's size bound includes entries a retirement receipt pins Final review of #8654 (P3): entries pinned by an outstanding retirement receipt (#8416, ownsRetirement) are kept past retention and the settled cap, at most two per receipt, and receipts have no cap of their own. The comment on OPERATION_JOURNAL_MAX_RECENT_SETTLED and structure/remote-link.md now include them in the stated file size. * fix(link): link_retirement_owned offers only the retries that can finish a journal-owned removal Lane review (P2): when a journalled operation owns the receipt, foreignRetirement refuses a revoke without --force that is not its id, whether or not the Child cleanup can run, so 'revoke without --operation-id (add --force if its cleanup cannot run again)' offered a retry that cannot finish. The text now says: repeat the owning operation id (ocx link operations lists the link's revokes) or ocx link revoke --link-id <link id> --force --yes without --operation-id; a revoke without --operation-id and without --force finishes it only when no journalled operation owns it and its cleanup can run again. The tombstone_limit hint names --force --yes too. * fix(link): operation_journal_unreadable makes no state claim on a journalled revoke Final review of #8654 (P2): a repeat of a journalled revoke whose own retirement receipt is outstanding resumes the removal first (#8416); if that finishes and the replay's journal read then fails, the answer is operation_journal_unreadable while the link is already gone, and 'Nothing was changed; ... retry' sent the caller into link_not_found with exit 4. The text now makes no state claim, names the way out (keep a copy, move it aside, repeat with the same id) and says that a link_not_found then means the link is already gone. The 'changes nothing' lists in structure/remote-link.md and the constant's comment no longer include an unreadable journal or a 5xx. * docs(link): the structure doc describes the tombstone_limit advice the CLI now gives Final review of #8654 (P3): structure/remote-link.md still said the tombstone_limit text tells the caller to keep the original id when a new one answers link_retirement_owned, the advice 08ce109 removed because it could loop. It now describes the current text: an unfinished removal holds the link; repeat the revoke that owns it, or finish with ocx link revoke --link-id <link id> --force --yes without an operation id. * fix(link): name the owner of an unfinished removal by what the CLI can show Final review of #8654 (P3): 'repeat the operation id that owns it (ocx link operations lists this link's revokes)' pointed at a list that shows no link id and marks no owner; the owner id is only in the server's message, which the CLI drops. The tombstone_limit and link_retirement_owned texts now say: repeat the open or unknown revoke of this link (ocx link operations --json shows each entry's linkId), or finish with a plain forced revoke. A closed owner field is a later change. * fix(link): generic revoke refusals name a next step, and an unknown outcome repeats before resolving Final review of #8654 (P3): link_changed and operation_journal_failed printed a bare 'Link revoke did not complete (code ...)'. They now add a next step that holds on every path that answers them: repeat the revoke with the same operation id (for operation_journal_failed, a new id if the same code comes back; for link_changed, after reading ocx link status). For an unknown outcome the CLI said 'check the Child, then resolve', skipping the repeat that finishes a removal this id still owns (#8416); it now says to repeat with the same id first and only then check the Child and resolve. * fix(link): the mode-mismatch text makes no state claim and offers no retry that answers operation_mismatch Final review of #8654 (P3): 'interrupted after its cleanup ran' claimed a state that is false for a force-mode receipt, and 'or with --force' with the same operation id answers operation_mismatch. With --operation-id the code is unreachable at this head (foreignRetirement admits only this operation's own receipt, resumed in its own mode), but the closed text now says only that the removal was interrupted in another mode and offers the revoke that started it with its options, or ocx link revoke --link-id <link id> --force --yes without --operation-id. * fix(link): the tombstone_limit message and the guide no longer say the link and key stay active Final review of #8654 (P3): the clause the CLI dropped in 74bf9a2 survived in the server's tombstone_limit message and in the recover paragraph of both guides. A repeat of an interrupted forced revoke can meet the limit after its earlier attempt already revoked the key and stopped the tunnel. The server message now says only that this attempt was refused before its own effects and names ocx link recover --link-id/--list; the guide (English and Spanish) says a fresh refusal changes nothing and to check ocx link status after a repeated one. * docs(link): resumes rather than finishes, a qualified pin bound, a valid recover command and --link-id in the hint Final review of #8654 (P3 nits): a revoke whose own removal was interrupted resumes it first (the resume can fail), in the guides, the capability text and its generated reference, structure/remote-link.md, the constant's comment and the CLI; the receipt pin is at most two entries per readable receipt and every revoke entry of a link whose receipt cannot be read; the tombstone_limit text names ocx link recover --link-id <id> (bare ocx link recover is a usage error) and the full repeat command with the link id the CLI knows; the describe title of the CLI tests now covers every refusal it holds. * docs(link): the truth table every journalled-revoke refusal text follows Final review of #8654 asked to change the method: structure/remote-link.md now holds the table, per code a journalled revoke can answer, of where it arises, what may already have changed on a first and on a resumed attempt (#8416, #8661), and what the CLI prints. A text claims a state only when it holds in every row for its code; journalledRevokeError's comments point here. * fix(link): the owner of a removal can be in any state, and operation_journal_failed keeps the same id Lane review (P2, P2, P3): a failed_clean revoke can own a retirement receipt too, so the tombstone_limit and link_retirement_owned texts now say to repeat the revoke of the link that owns it, whatever its recorded state (another revoke's id answers its own outcome or the same refusal). A resumed owner can meet the same live journal-write failure again and keep its receipt, so operation_journal_failed now says to fix the journal's disk and repeat the same id, and that a new id, if tried and answered with link_retirement_owned, means this id may still own the removal. The table's tombstone_write_failed row now says the write's outcome is uncertain: recordTombstone can throw after its atomic write, before this revoke counts the tombstone as its own. The table in structure/remote-link.md follows all three. * fix(link): link_changed says what a second answer means instead of claiming the link changed during this revoke * fix(link): an unknown revoke resumes a removal on repeat only if its id still owns one * fix(link): revoke and recover commands in the guidance carry the link id so they run as printed * docs(link): the truth table says what operation_journal_failed and a resumed tombstone_limit can leave behind * fix(link): an add refusal says only that the apply changed nothing, since confirm-host already pinned the host key * fix(link): the tombstone_limit message and the guide name only what the refused attempt left alone * fix(link): an unknown revoke resolves only once its entry reads unknown or residual; an open one is repeated * docs(link): the truth table tells a replayed link_changed from a resumed one and names what a first-intent failure leaves
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This follows up the operation journal from #8623 (
ocx link addandocx link revokeunder--operation-id). It fixes the nine small findings from that PR's final review. It also closes one real gap: a journalled revoke that may already have purged or disconnected its Child could later reconnect the tunnel to it, invisibly.A link stays down after an uncertain revoke, and says so. A journalled revoke can stop
unknownafter it reached the Child: a purge receipt that could not be written, a key revoke or record removal that failed after the cleanup, a timeout, or a purge with nopurged:truereport. Before this change, the next reload or Home restart started the tunnel again. Now the supervisor starts no tunnel for a link whose revoke has written itsremote_cleanupintent and either settledunknownor is still open with no revoke in this process running it, which means the Home stopped during it (revokeSuspendsLink, checked inspawnFor).failedwith reasonrevoke_unsettled, since the revoke settled. The row keeps its keys, so every CLI's strict status check accepts it.ocx link reconnectand the dashboard's Retry stop nothing and answer409 link_suspendedwith the operation id and state. The CLI builds the next step itself from the validated id and exits 5.ocx link operations resolve --operation-id <id> --yesreloads the tunnels and the link reconnects. An open entry cannot be resolved; repeat the revoke with the same--operation-idand--purgeflag first.A journalled revoke still restarts the tunnel itself, and settles
failed_clean, when the Child's disconnect provably never ran: platform discovery failed, the purge's runtime inspection failed or found a full install, or there was no planted binary to run. A running revoke counts as live, so its own restart goes through. One that leaves the tunnel down drops its live mark just before it releases the supervisor hold, so a reload before the outcome is written already finds the link suspended. A damaged journal suspends nothing.The journal cannot lock a Home out. Only unsettled entries (open, and unresolved residual or unknown) refuse a new operation, at 256, and the message names the way out. Settled entries past 256 are pruned oldest first. None is pruned before
OPERATION_PRUNE_FLOOR_MS: a day, plus the 120 s alias-lock wait, plus a minute. So a retry that starts within a day of an answer it lost still replays it. A new operation is otherwise refused only after 1024 operations settled within that floor, which holds a runaway loop for at most a day instead of 30. The file holds about 512 entries (about 0.5 MiB), at most about 1.5 MiB in such a burst.The nine follow-ups from #8623's review:
operations.json.exclusiveis out of the supervisor's record instance.operationIdwithout a fingerprint.planted: true.alias-lock.tshas the real-home guard.--forcestill goes through.What stays unchanged by default
A Home that never journalled is unchanged. The journal file exists only after a caller passed
--operation-id, and nothing does until GOAT does. On such a Home, each tunnel spawn attempt and each status read costs oneexistsSync. Plain revokes write no entry and restart the tunnel after a failed cleanup as before.Once a Home has journalled, plain callers there see four things:
link_suspendedreconnect andrevoke_unsettledstatus for a link an unsettled journalled revoke keeps down;--forcerunning past a damaged journal, without the open-entry check;An
unknownrevoke entry already written by a #8623 build keeps its link down after the upgrade until it is resolved. No release contains #8623.Planted: this branch's CLI accepts every enrollment answer shape a Home of this release line sends:
{linkId}without an operation id, and{linkId, operationId, outcome}with or withoutplanted: true. The server sendsplantedonly on journalled answers, neverfalse. Tests cover all three shapes.Enrollment deadline (checked, not changed): the worst case does not fit the 240 s left after a full 120 s lock wait under the CLI's 360 s deadline. It does not fit 360 s either. After the lock, apply's steps can each run to their own timeout. Here B is the SSH command budget: min(60 s, 15 s + 4 × measured latency).
That totals 225 s + 7 × B: 330 s at zero latency and 645 s at the 60 s cap. This predates this branch. A typical apply takes seconds per step. When the CLI's deadline passes first, it prints its existing "outcome is uncertain" message. With
--operation-id, a repeat replays the journalled outcome, or waits on the alias lock and then replays. Matching the budgets to the deadline is left for a separate change.Verification
Every logic change was test-first: the new test failed before its fix for the stated reason.
exclusive-only restart.Commands on head 007b1e9 (bun 1.4.4,
TMPDIR=/tmp):bun run typecheck→ exit 0bun run privacy:scan→ exit 0bun run structure:check→ exit 0bun run skill:surface:check→ exit 0bun scripts/test.ts --parallel=2over tests/clients/link-, tests/server/link-, tests/cli/cli-link*, the test-layout, layout-tooling, file-size ratchet, structure, skill-ocx, docs-spanish-localization, docs-remote-hub-claims and docs-link-targets tests, and tests/lab/core-link-boundary and core-lab-boundary (126 files) → exit 1: 2104 pass, 23 skip, 3 fail. The 3 failures are the Rust-toolchain cases under NOT RUN; they fail the same way on dev.git merge-tree --write-tree origin/dev HEAD→ exit 0 against dev 6b36883.Merges of dev: two normal merge commits.
src/link/supervisor.ts. I kept dev'sCANDIDATE_KILL_GRACE_MS, createRouteRounds, type LinkRouteStatusimport and added this branch's operations-journal import after it. Everything else merged on its own. A suspended link whose child is draining is still escalated and forgotten when it exits, andspawnForthen declines it. A revoke's hold thenstopLinknow ends within about 2 s for an ssh that ignores SIGTERM. Both supervisor test files andtests/clients/link-revoke-suspend.test.tspass on the merge result (133 pass).The Spanish guide mirrors the new paragraph's code spans, and docs-spanish-localization passes.
NOT RUN:
bun run testsuite (this machine runs focused files only);tests/clients/link-ocx-child-e2e.test.ts(2 cases) and the Rust conformance runner intests/clients/link-ocx-child-conformance.test.ts, which need a Rust toolchain this machine lacks;gui/change: Retry on a suspended link shows the generic error, and the row shows the generic reason label with coderevoke_unsettled; a localized string belongs to the dashboard follow-up, since agui/change needs a screenshot in this PR);Review: an independent read-only reviewer covered this branch. One reviewer took rounds 1 and 2; a fresh reviewer, briefed with the earlier findings, took rounds 3 to 8. Rounds 6 to 8 covered this hand-over's changes on top of 27572ec. Across all rounds it raised 8 P2 and 3 P3, each fixed in its own commit. Round 8 reported no open P0-P2. Its P3 was a wording fix (30887f9), and that commit and the clean merge 007b1e9 came after its last round. It ran only the focused revoke, suspension, journal, enrollment and CLI test files.
Docs:
structure/remote-link.md"Operation journal" covers the suspension, its visibility and way out, and the journal bounds.skills/ocxsurface describelink_suspended, the reload on resolve, the way out for an open revoke and the bounds.Checklist
devsrc/have focused regression tests in their domain folders, registered in the test layout maps