Skip to content

fix(auto-reply): allow the Flash-Next Sushi model and survive EPERM in the group sweep - #57

Merged
twoimo merged 6 commits into
mainfrom
fix/flash-next-sushi-and-killpg
Nov 16, 2026
Merged

twoimo merged 6 commits into
mainfrom
fix/flash-next-sushi-and-killpg

Conversation

@twoimo

@twoimo twoimo commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Why

The owner wants auto-reply running again for one room on the local Qwen3.8 Flash-Next model. Their on-demand MLX proxy serves it as beamster/Qwen3.8-Flash-Next-Sushi-2bpw. Startup refused that ID. In 1.9.0 the session guardian also crashed with PermissionError: [Errno 1] Operation not permitted when it swept the process group of a leader that had already exited.

What changed

  • One table of allowed local models now drives both runtimes: LOCAL_MLX_MODELS in Rust and LOCAL_REPLY_MODEL_FAMILIES in Python. Each entry has an exact ID, which can take an mlx/ prefix, and a Flash-Next or 27B family. The Sushi ID is a Flash-Next entry. The local reply choice carries its table entry, so the requested ID survives apply() and the readiness probe.
  • The worker uses the table for its local-model, 27B and gateway-marker checks. Its response-model check now rejects an answer from another model to a Sushi request.
  • _terminate_owned_child treats EPERM from killpg like ESRCH, so the guardian returns the child's exit code and does not crash.

Scope

  • Covers the CLI startup gate, the readiness probe, the worker's generation path and the guardian's group sweep.
  • Leaves out the menu-bar model picker, the Jarvis desktop lists, the model-swap manager and fine-tuning. These still name only the two MLX-Serve models.
  • Source-only. The Python changes reach production only after a new runtime is baked and installed.

Tradeoffs

  • The worker's Flash-Next and 27B checks are now exact lookups, not case-folded substring matches. A non-exact Qwen3.8 variant can come only from a menu-bar override file. The worker refuses the bare form. The mlx/ form takes the same path as any other mlx/ model.

Blast Radius

The change touches every local MLX auto-reply startup and generation. A differential check shows the two existing MLX-Serve IDs classify as before in every form. Startup still refuses every other local ID, including the Qwen3.6 35B. The guardian change affects only shutdown after EPERM. A member that outlives the leader still gets SIGTERM, then SIGKILL.

Verification

  • Each fix has a test commit that fails, then a fix commit that passes. The guardian test reproduced the live traceback, run_session_guardian into os.killpg(..., SIGTERM).
  • cargo fmt --check and cargo clippy --all-targets -- -D warnings are clean. cargo test passes 1,172, up from 1,169.
  • The CI Python list, run with an empty HOME under a write guard, passes 1,372 with 9 skipped, up from 1,366.
  • Not run against the live proxy. The probe needs the Sushi row to report loaded: true and state: "ready". The worker needs owned_by: "mlx-serve" or an mlx/ ID.

🤖 Generated with Claude Code

twoimo and others added 6 commits November 16, 2026 21:34
…reply model

The owner's local MLX proxy serves beamster/Qwen3.8-Flash-Next-Sushi-2bpw.
Startup refuses it today, and the worker falls back to the MLX-Serve
Flash-Next ID or sends an mlx/-prefixed form through the unchecked opener.

These tests require the Sushi ID, bare or mlx/-prefixed, to pass startup
validation, keep its identity through the readiness probe, generate as a
Flash-Next text model through the loopback gateway, and reject an answer
from another model. Other local IDs, such as the Qwen3.6 35B, stay refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cal Flash-Next model

The owner's local MLX proxy serves only the Sushi build of Qwen3.8
Flash-Next. Startup refused it, and the local choice rewrote any
Flash-Next request to the MLX-Serve ID, which the proxy does not
advertise.

Both runtimes now read one table of allowed local models, keyed by
exact ID with an optional mlx/ prefix, each with a Flash-Next or 27B
family:

- Rust: LOCAL_MLX_MODELS in config.rs. AutoReplyLlmChoice::LocalMlx
  carries the table entry, so apply() and the readiness probe keep the
  requested ID. The interactive menu lists each local model by ID.
- Python: LOCAL_REPLY_MODEL_FAMILIES in auto_reply_ondevice.py. The
  worker's gateway-model, 27B and gateway-marker checks read it, and
  the response-model check now catches another model answering a Sushi
  request.

Startup still refuses every other local ID, including the Qwen3.6 35B.
The worker's existing rule for mlx/- and omlx/-prefixed IDs, and the
menu-bar, Jarvis desktop and model-swap lists, are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eader's group

On macOS, killpg can return EPERM for a group whose leader has exited
when no member is left that the guardian may signal. In 1.9.0 the
session guardian then crashed in _terminate_owned_child with
PermissionError: [Errno 1] Operation not permitted.

These tests use a fake os.killpg. The guardian must return the child's
exit code when every killpg call returns EPERM. The sweep must signal
a member that outlives the leader. It must stop without SIGKILL once
the group reports EPERM, and survive EPERM from the final SIGKILL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the session guardian's leader exited, _process_group_exists read
EPERM from killpg(pgid, 0) as "a member remains". The sweep then sent
SIGTERM, which raised PermissionError and crashed the guardian.

killpg reports EPERM when no member can be signalled. macOS reports it
for a group whose leader has exited. The probe now treats EPERM like
ESRCH. A SIGTERM or SIGKILL that fails with EPERM no longer raises. A
failed SIGTERM ends the sweep only when the probe confirms no member
remains. A member that outlives the leader still gets SIGTERM, then
SIGKILL at the end of the grace period, and the sweep stops signalling
a group ID once it reports empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker picks the first loaded gateway row whose ID contains the
requested model's family name, sends the request there, and checks the
answer against that substituted ID. These cases pin the requested
model instead:

- With ddalcu Flash-Next and Sushi both loaded, in either order, each
  request goes only to the model it names, and an answer from the other
  one is rejected.
- With an unlisted Sushi-4bpw or 27B-8bit row listed first, the Sushi
  and 27B requests still go to the allowed ID.
- When the requested model is not loaded, the request fails closed with
  mlx_serve_text_model_not_advertised instead of reaching a lookalike.

At this commit 12 subtests fail. A Sushi request with only ddalcu
loaded returns (0, b'ok', b'') from ddalcu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker chose the gateway row by family name. It took the first
loaded row whose ID contained "Qwen3.8-Flash-Next" or "Qwen3.8-27B",
sent the request there, and checked the answer against that row. A
Sushi request could go to ddalcu Flash-Next or to an unlisted
Sushi-4bpw build and pass the answer check. A ddalcu request could
likewise go to Sushi.

loaded_local_reply_model_id now selects only a loaded row whose
canonical ID equals the requested model's canonical ID. If there is no
such row, the call fails closed with mlx_serve_text_model_not_advertised,
or mlx_serve_vision_model_not_resident for a 27B image request. The
answer check compares the response's model with the requested model.
The payload still carries the row ID as the gateway lists it.

The diagnostic probe and recommend_ondevice_setup still match by family.
They report on the gateway and never route a reply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@twoimo
twoimo merged commit cf08d55 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant