PR #304 IPC: daemon restart regresses over UDS on ubuntu CI ("restart within 45s"); passes over TCP on main

[StefanSteiner]

Symptom

On PR #304 (feat(mcp): connect the daemon to its engine over IPC), two of the three daemon restart tests fail on test (ubuntu-latest):

thread 'engine_recovers_after_hyperd_killed' panicked at hyperdb-mcp/tests/daemon_tests.rs:1744:9:
daemon should restart hyperd within 45s
thread 'hyperd_monitor_detects_killed_hyperd_and_restarts' panicked at hyperdb-mcp/tests/daemon_tests.rs:1647:9:
daemon should restart hyperd within 45s
test result: FAILED. 61 passed; 2 failed

client_report_triggers_restart_after_kill (the third restart test) passes. This is after #309 raised the readiness budget 12 → 45s, so it is not the old budget-too-small timeout — the daemon genuinely does not restart hyperd within 45s over UDS under CI load.

Why this looks like a real UDS regression, not the known flake

The distinguishing evidence: the same two tests pass on main (TCP transport) on ubuntu — confirmed green at 82b45cb (the rc.3 release commit) and on #309's CI run (bea4855, test (ubuntu-latest) passed). They fail only on #304, whose sole functional change is switching the daemon's engine connection from TCP to a Unix domain socket. So the delta is the transport, not test flakiness or CI noise.

They pass 4/4 locally (macOS dev box, the adversarial reviewer's runs, and the fix-worker's runs), so the failure is load/timing/platform-dependent and does not reproduce on a developer machine — which is why it slipped past local gates and the review.

Likely mechanism (from the #304 review's own I-1/I-4 findings)

hyperd refuses to bind an already-occupied domain-socket path and dies ("domain socket is in use"). The restart path works only because try_restart_hyperd drops the old HyperProcess (reaping the SIGKILLed child) before respawning, so hyperd's pid-liveness staleness check on <dir>/hyper.pid lets the replacement rebind the same socket path. Under ubuntu CI load, the reap-then-rebind sequence appears to race — the stale socket/pid state may not clear in time, so the rebind is refused or delayed past 45s and the restart never completes. The fast-fail pre-flight added in the fix cycle only rejects an already-bound socket; it does not fix a slow or racy rebind.

Over TCP (the previous transport) this never happened because each restart bound a fresh ephemeral port, sidestepping the stale-endpoint problem entirely.

Impact

#304 must not merge until this is resolved. It is a reliability regression in the daemon's core value — crash detection and restart — over the new transport. It is not urgent: rc.3 is already shipped, rc.4 is not being cut, and #304 is future work. But it blocks #304, and it is a genuine finding about the IPC design, not a test artifact.

Next steps

1. Confirm consistency — re-run feat(mcp): connect the daemon to its engine over IPC #304's ubuntu job a few times. A flake would intermittently pass; a real rebind race should fail most runs. (One re-run is in progress as this is filed.)
2. If consistent, fix the UDS stale-socket rebind on restart: likely try_restart_hyperd (or hyperd startup) should explicitly unlink the stale socket path before rebinding, rather than relying on the pid-staleness check racing the reap under load. This is the same territory as the review's I-1 and the deferred single-instance-lock work.
3. Reconcile with Decide whether the 8 macOS-skipped daemon-mode tests should be re-enabled (or the guard made explicit) #305 (the macOS-ignored restart tests) and Flaky: scan_all_refused_returns_freeport_base sees another test's fixture discovery record #300 (the shared-state flake) — the restart-test reliability story now spans TCP-flaky-on-macOS and UDS-broken-on-ubuntu.

Provenance

Surfaced when #304 was update-branched onto main (with #309's fixes) and its test (ubuntu-latest) leg failed 2/3 restart tests. Verified the same tests pass on main over TCP. #304 head e51fdefb, run 34092465671.

[StefanSteiner]

Update — the re-run passed, so this is intermittent, not a deterministic regression.

Re-running #304's failed test (ubuntu-latest) job (same head e51fdefb, run 34092465671) came back green — both engine_recovers_after_hyperd_killed and hyperd_monitor_detects_killed_hyperd_and_restarts passed on the retry.

So the title's "regresses over UDS" is too strong. What we actually have:

• On feat(mcp): connect the daemon to its engine over IPC #304 (UDS): the two restart tests failed once ("restart within 45s") and passed on re-run — flaky.
• On main (TCP): they passed (limited data — one or two runs).

This is most likely the same known ubuntu restart-test flakiness tracked in #305 and #300, rather than a UDS-specific determinism — though UDS's tighter socket-rebind timing (vs. TCP's fresh ephemeral port per restart) is a plausible reason it would flake more often over UDS. Establishing whether UDS is genuinely more flaky than TCP needs multiple runs on each transport, not a single pass/fail each.

Net: not a hard blocker in the sense I first wrote, but a real reliability question on the IPC change's restart path that should be understood before #304 merges — merging it as-is means these restart tests will intermittently red main's ubuntu leg. Folding this into the #305 restart-test reliability decision is probably the right home.

[StefanSteiner]

Correction to my previous comment — the re-run did NOT pass. It failed again, so this is reproducible, not intermittent.

I misread the re-run: I took a shell pipe's exit code as the run's result. Reading run 34092465671's actual job conclusions directly:

test (ubuntu-latest): completed/failure  (attempt 2)
  engine_recovers_after_hyperd_killed ... FAILED
  hyperd_monitor_detects_killed_hyperd_and_restarts ... FAILED
  "daemon should restart hyperd within 45s"
  test result: FAILED. 61 passed; 2 failed

So both attempts (1 and 2) failed on ubuntu over UDS with the identical two restart tests and the identical symptom. This is consistent/deterministic, not the flake my prior comment claimed. The original framing of this issue stands: the daemon does not restart hyperd within 45s over UDS on ubuntu CI, reproducibly, while the same tests pass over TCP on main.

Please disregard the "intermittent, not a hard blocker" characterization in the comment above — that was based on the bad exit-code read. This is a real, reproducible restart regression on the IPC/UDS path and a hard blocker for #304.

Two data points now (both #304 attempts red on ubuntu-UDS; main green on ubuntu-TCP) — strong enough to treat as a genuine UDS restart bug, most likely the stale-socket rebind race described in the issue body. #304 stays held until it's fixed.

[StefanSteiner]

Root cause CONFIRMED — and it is not the stale-socket rebind hypothesis in the issue body. That theory is refuted.

Two agents investigated independently and converged on the same answer, measured directly on the failing ubuntu runner rather than inferred.

The bug is in the test harness. find_hyperd_pid_for_endpoint located the engine with:

lsof -nP -t -c hyperd <socket path>

lsof ORs its list-selection options unless -a is passed. So that query returned every hyperd process on the runner — not the one owning the socket — and -t prints ascending PIDs, so the helper returned the lowest PID. The test therefore SIGKILLed an older test's engine while its own stayed alive. Phase 1 of wait_for_live_hyperd_after_kill ("the killed endpoint must stop accepting") consequently never satisfied, and the wait burned its entire budget.

Measured on the runner:

endpoint=/tmp/.tmpwYJDzM/sockets/hyper
pid_file_raw=Ok("17736")                      <- this endpoint's engine
OR_form  (-c hyperd <path>) -> "17675\n17736"  <- wrong PID returned first
AND_form (-a -c hyperd <path>) -> "17736"      <- correct
  17675 hyperdMain  <- an OLDER engine, still shutting down
  17736 hyperdMain  <- this test's engine

Two engines overlap because TestDaemon::drop waits only for the health port to close while run_daemon drops the HyperProcess afterwards, so the next test's engine starts before the previous one has exited.

This explains every observation that puzzled us:

• Deterministic, not flaky — the wrong PID is selected every time. The two failures took 44.8s and 45.4s: exactly the budget, which is why test: make changelog contract robust to rollover; bump ubuntu restart budget #309's 12s→45s bump changed nothing.
• Transport-specific — the TCP branch used -iTCP:<port> -sTCP:LISTEN, which names exactly one listener because the port is unique. Only the socket-path branch added in feat(mcp): connect the daemon to its engine over IPC #304 is ambiguous. So feat(mcp): connect the daemon to its engine over IPC #304 did cause it, via its test helper, not its production code.
• Why the third restart test passes — client_report_triggers_restart_after_kill sends REPORT_HYPERD_ERROR, and the monitor honours that flag even when the engine is alive (run.rs process_dead / consume_restart_request), so it restarts regardless of whether the kill hit the right target. It never depended on kill accuracy — exactly the asymmetry the OR bug predicts.

Why the rebind hypothesis is out: no "domain socket is in use" appears in either failing log; hyperd fs::removes a stale socket itself before binding (hyperd.cpp:552); and HyperProcess::has_exited() calls try_wait(), reaping the SIGKILLed child at detection time — a full monitor tick before the respawn — so the old PID is ESRCH and hyperd clears its own stale pid file. Empirically the same fixed socket path now rebinds in ~5s, repeatedly. No production change is needed.

Fixes

Both independent attempts fix the same function:

• On feat(mcp): connect the daemon to its engine over IPC #304 (feat/daemon-engine-ipc, head 0516aa56): add -a so lsof ANDs its selectors. Minimal; independently verified correct by the other agent's runner measurement above.
• On fix/uds-restart-linux-opus (head 1e5b78d, not pushed to feat(mcp): connect the daemon to its engine over IPC #304): read the PID from hyperd's own lock file (<endpoint>.pid — provable from Display for ConnectionEndpoint::DomainSocket and hyperd's PIDFile), confirmed with ps -o comm=, removing the lsof dependency and its OR/AND footgun entirely. Also makes the readiness helper return an error naming which phase expired. 3 consecutive fully-green test (ubuntu-latest) runs; restart tests green 6/6, completing in ~5-6s instead of burning 45s.

Taking the -a fix on #304 to unblock it. The PID-file approach and the phase-naming diagnostics are worth folding in as a follow-up — the opaque "didn't restart in time" message is precisely why this took two CI runs and two investigations to identify.

[StefanSteiner]

Resolved. Root cause was the lsof OR-selection bug in the test harness (full analysis above, not the stale-socket rebind theory this issue originally proposed). Fixed on #304 by adding -a so lsof ANDs its selectors; #304 merged as bcd88fd with all legs green — test (ubuntu-latest), test (macos-14), test (windows-latest), and both clippy legs.

The restart tests now pass on Linux over UDS, completing in ~5-6s instead of burning the 45s budget.

Follow-up hardening (read hyperd's own pid file instead of shelling out to lsof, and name the expired readiness phase) tracked in #311.

[StefanSteiner]

Resolved — root cause found, fix merged in #304 (head 0516aa5)

The leading hypothesis in the issue body ("hyperd refuses to bind an occupied UDS path; the reap-then-rebind races under load") is refuted. The real cause is a test-harness bug, not a daemon/hyperd defect: the restart tests were killing the wrong hyperd process under parallel load.

Confirmed root cause

find_hyperd_pid_for_endpoint (in hyperdb-mcp/tests/daemon_tests.rs) located the hyperd to kill over a Unix domain socket with:

lsof -nP -t -c hyperd <socket_path>

lsof combines separately-stated selectors with OR, not AND, unless -a is given (confirmed against man lsof: "Normally list options that are specifically stated are ORed", and empirically: lsof -c sleep /etc/hosts returns the union of every sleep pid and every holder of /etc/hosts, while lsof -a -c sleep /etc/hosts returns nothing). So -c hyperd <path> meant "every process named hyperd, OR anything holding <path> open" — i.e. the pids of all concurrently-running hyperd processes, not the one bound to this test's socket. .find(first) then returned an unrelated test's hyperd.

Sequence of the failure:

1. Test kills a pid from that OR-union → an unrelated test's hyperd.
2. This test's own hyperd stays alive.
3. The daemon's liveness monitor correctly sees hyperd alive → never restarts (correct behavior).
4. wait_for_live_hyperd_after_kill Phase 1 waits for the killed endpoint to stop accepting connections; it never does (the real hyperd is up) → 45s timeout.

Why UDS-only / why a #304 regression: the TCP branch selects the listener with -iTCP:{port} -sTCP:LISTEN, which names a unique listener even under OR semantics — so the tests always passed on main over TCP. #304 switched the daemon endpoint to a UDS path, routing pid-discovery down the new -c hyperd <path> branch and exposing the OR bug. Why serial/macOS passed: with only one hyperd alive at a time (serial run, or macOS where these three tests are #[ignore]d), the OR-union collapses to that single correct pid.

Evidence (verified, not assumed)

• Failing run 34092465671 (e51fdefb, before fix): captured in-process daemon tracing + hyperd log tail on timeout showed the original hyperd still alive and accepting connections at the 45s mark, only one hyperd log file, and no restart attempt in the daemon trace. There is no "domain socket is in use" anywhere — hyperd never tried to rebind, because it was never killed. This directly refutes the rebind-race hypothesis.
• lsof OR-vs--a semantics confirmed from the man page and empirically on this host.
• Local: the failure reproduces in the parallel full daemon_tests binary and never with --test-threads=1 (serial), consistent with the multiple-hyperd OR-union mechanism.

The fix (test-only)

Add -a so the command-name and path selectors are ANDed, returning only the hyperd actually bound to the socket path:

lsof -nP -t -a -c hyperd <socket_path>

Chosen over the three daemon-side directions (unlink-stale / confirm-reap / fresh-socket-path) because none of those address the actual cause — the daemon and hyperd were behaving correctly; the harness was killing the wrong process. This is a one-selector, test-only change (no production or public-API change, so no changelog entry per the per-crate policy). An earlier speculative daemon-side "clear stale UDS artifacts before restart" commit + temporary tracing diagnostics were dropped (branch reset) once the trace disproved the rebind hypothesis, keeping #304 minimal.

CI evidence (read from JSON conclusions + real log output)

Fix commit 0516aa5. test (ubuntu-latest) — the leg that failed 2/2 before — went green 3 consecutive times (run 34101597585, original + 2 reruns), each with all three restart tests confirmed in the log:

test client_report_triggers_restart_after_kill ... ok
test engine_recovers_after_hyperd_killed ... ok
test hyperd_monitor_detects_killed_hyperd_and_restarts ... ok
test result: ok. 63 passed; 0 failed

Plus 8/8 consecutive green parallel runs of the full daemon_tests binary locally (63/63 each).

Confidence: high. Unlike a load-timing patch, this fixes a deterministic selector bug with a mechanism confirmed from first principles (lsof semantics) and matching CI/local evidence on both sides of the change. Merged as the head of #304 (0516aa5).