Orchestration benchmark harness and the engine observation seam - #750
Open
tony wants to merge 67 commits into
Open
Orchestration benchmark harness and the engine observation seam#750tony wants to merge 67 commits into
tony wants to merge 67 commits into
Conversation
tony
force-pushed
the
engine-ops-orchestration-bench
branch
2 times, most recently
from
August 23, 2026 01:28
7d65eb8 to
a28598c
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## engine-ops-hardening #750 +/- ##
========================================================
+ Coverage 73.56% 73.94% +0.38%
========================================================
Files 172 173 +1
Lines 11809 11913 +104
Branches 1898 1914 +16
========================================================
+ Hits 8687 8809 +122
+ Misses 2449 2439 -10
+ Partials 673 665 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 13:31
462044d to
ed92fb3
Compare
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 15:36
05b1613 to
8375633
Compare
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 15:58
8375633 to
5371eec
Compare
why: Define truth and safety contracts before a benchmark that can create 40,000 live panes. what: - Separate destructive setup from repeatable active-topology phases - Specify deterministic Rich workloads and resource-aware ramping - Define correctness, raw-sample, resource, and cleanup evidence
why: Repeated wait samples need distinct emission evidence without restarting the active topology. what: - Define run-scoped sentinel request and evidence markers - Add supervisor recovery and explicit resource floor contracts
why: Give the orchestration benchmark deterministic active pane output without starting a Python process in every pane. what: - Add a paused, run-scoped stream service with atomic control markers - Add deterministic frame rendering and request-specific sentinel evidence - Cover gates, invalid markers, repeated waits, and graceful shutdown
why: Let benchmark reports separate configured wait delay from generator scheduling lateness and reject ambiguous control marker schemas. what: - Record request time, configured delay, and scheduling lateness in evidence - Require an exact integer schema version for every control marker - Add timing, schema, and executable documentation coverage
why: Keep the fuzzer's long-running public entry points verified as executable documentation. what: - Add temporary gated serve doctest - Add redirected Rich preview and CLI doctests
why: Keep large orchestration runs within observed host limits and retain valid evidence when they stop early. what: - Add immutable topology, resource guard, and report records - Provide atomic JSON validation and a tmux-free plan command - Cover cgroup probing, admission, summaries, and artifacts
why: Preserve durable and immutable benchmark evidence across resource failures. what: - Sync atomic report replacement and freeze public mappings - Retain independent host probes and validate verified samples - Bind ramp attempts to declared topology sequences
why: Prevent contradictory ramp artifacts from being accepted as benchmark evidence. what: - Add explicit single, canonical, and custom ramp state - Bind terminal attempts to declared sequence outcomes - Cover contradictory and noncanonical ramp reports
why: Report validation must reject contradictory runtime ramp data. what: - Validate ramp kinds and attempt status vocabulary - Enforce one terminal transition with matching later reasons - Add invalid discriminator coverage
why: Runtime artifact types and generated API docs must reject invalid state and describe each callable accurately. what: - Validate report and guard decision discriminator vocabularies - Cover exact terminal ramp and shape declaration state - Add NumPy sections and doctests across benchmark callables
why: In-progress ramps could retain terminal outcomes, completed work past pending entries, or reasons on unattempted shapes. what: - Enforce the completed-prefix and pending-suffix checkpoint grammar - Require terminal attempts to finalize the report lifecycle - Cover malformed and valid in-progress ramp sequences
why: Exercise every engine lane against an exact active tmux topology without contacting ambient servers or leaking owned processes. what: - Build and stabilize one typed WorkspaceSet across all four lanes - Verify exact snapshots, activity epochs, and process identities - Clean fuzzer, engine, server, followers, socket, and scratch state
why: Partial setup failures could lose cleanup evidence, permissive directories exposed run state, and exact counts did not prove parentage. what: - Reap pre-return children and surface typed cleanup evidence - Enforce private scratch and bounded owned socket paths - Verify declared topology parentage with adversarial coverage
why: A chmod or stat failure after mkdir could bypass caller ownership flags and leak a scratch or socket directory. what: - Roll back newly created directories with exact Path.rmdir calls - Preserve acquisition and rollback failures in a typed exception - Cover scratch and socket-root chmod and stat failure paths
why: Measure active bulk mutation and query strategies against one persistent verified tmux topology. what: - Add typed mutation, enumeration, capture, and search phase results - Interleave repeatable strategies and reject invalid timing samples - Cover sync and async subprocess and control-mode phase parity
why: Prevent stale heartbeat, misattributed capture, or cancellation from yielding accepted phase evidence. what: - Require fresh post-restoration activity and mandatory postconditions - Shield async restoration and bind captures to exact pane targets - Exercise explicit search families across all live lanes
why: Quantify request-scoped delayed output without restarting the active topology or conflating the configured delay with waiter overhead. what: - Harden schema-v1 sentinel request identity and durable evidence - Add sync/async capture polling and decoded control-stream waits - Verify repeated timing, stale-token, drop, and subscription contracts
why: A blocked typed capture could outlive its request deadline, and unbounded terminal identity could evade exact capture matching. what: - Bound sync subprocess/control and async capture work to the request deadline without leaving owned work behind - Restrict sentinel components, record size, producer rate, and capture history to one documented terminal-safe contract - Cover timeout cleanup, pending cancellation, and maximum-token round trips on the repeated active topology
why: Make active benchmark execution recoverable and preserve validated evidence across completion, refusal, cutoff, cancellation, and failure. what: - Add public run and ramp commands with a hidden checkpointing worker - Add sequence-based supervision and exact identity-checked cleanup - Validate JSON artifacts and render descriptive Markdown summaries - Cover real CLI phase graphs, ramps, recovery, and every engine lane
why: Recovery must bind process ownership without PID races and remain durable through journal corruption or repeated cancellation. what: - Retain Linux pidfds for every published identity and drain identity deltas throughout bounded recovery - Make progress appends durable, fail closed on corrupt terminal tails, and preserve evidence until cleanup is proven - Shield lifecycle finalization through repeated cancellation and tighten phase and cleanup validation
why: The design promises optional ORM cells reading server.sessions, server.windows, and server.panes as a reference alongside the typed operations, and no such measurement existed. Without it the third comparison cannot be made at all, so there is no answer to whether the typed seam improves on the incumbent. Note the naming trap this does not resolve: the classic search family means tmux server-side format filtering, not the classic ORM, so search.classic.* never satisfied this promise. what: - Add enumerate_orm(), timing a classic Server hierarchy read and validating it through the same acceptance path as the typed cells - Require exact row agreement with the typed operations, which is the only reason the reference is comparable at all - Cover it live against a real topology at every hierarchy level The reference is deliberately not a fifth lane: Server reaches tmux through its own request graph whichever engine the run measures, so its timing must never be folded into an engine speedup claim.
why: enumerate_orm() existed but no run could request it, so the third comparison still could not be made from the command line. what: - Add --with-orm to run and ramp, threading the opt-in through the supervisor, the hidden worker, and the report - Record the choice on the artifact as orm, and derive the required phase graph, interleaving groups, and fuzzer service budget from that declaration rather than from a fixed 38-phase constant - Interleave the reference cells with the typed enumeration they are compared against - Raise the report schema to 3, since a validator must read the declaration to know which graph to require A run without the flag is unchanged, and a report that declares the reference must carry its cells for validation to pass.
why: The page still described the classic ORM as an unfulfilled intention and showed no way to run a lane comparison, which are now the two things a reader most needs. what: - Add paste-and-run blocks for the matrix supervisor and its re-render path - Describe how the ORM reference is requested, what it adds, and why it is a reference rather than a fifth lane - Warn that the classic search family means server-side format filtering, not the classic ORM, which is the trap that made this promise look kept - Pass --with-orm through the matrix supervisor to every cell
why: The notes recorded a diagnosis that measurement later overturned, and a wrong remedy derived from it. Leaving that in place would send the next reader down the same two dead ends. what: - Record that tmux flips the socket's execute bits on attach and detach, with mtime unchanged, so whole-mode comparison was the defect - Retain both superseded readings, the mtime race and the scratch-location guess, with the evidence that refuted each - Describe the shipped ORM reference and the naming trap that hid its absence - State the per-phase claim boundary and the percentile gate, and reduce the outstanding list to what is genuinely left
why: Bootstrapping the retained fifteen-sample cell showed that raising the sample count does not rescue the marginal per-phase ratios. The median's spread barely moves between ten and fifteen samples on the noisy phases -- bulk mutation stays near 85 percent, session enumeration near 43 -- because the variance is run-to-run, not sampling noise that averages away. So a differences of 1.1x to 1.5x between lanes is not resolvable at any practical sample count, and reporting one as a number invites a reader to quote it. Only effects far outside the spread, such as the 142x on bulk mutation, survive. what: - Add median_interval(), a distribution-free order-statistic interval, since phase timings are heavy tailed and a deviation-based interval understates the spread - Report a phase ratio only when the fastest and slowest intervals separate, and mark it unresolved otherwise - Treat fewer than six observations as unable to bound a median at all Re-rendering the retained two-sample matrix now claims nothing and marks all 36 phases unresolved, which is the correct verdict for that evidence.
why: Scale limits and lane comparison are different questions and were being answered by the same oversized run. Comparison needs enough samples to separate medians; pressure testing needs none, because only the outcome matters. Conflating them produced a matrix that would have run for hours. what: - Add scripts/orchestration_stress.py, escalating one topology dimension at a time so a failure is attributable to panes, windows, or sessions - Run each rung at a single timed invocation with no warmup, and record the phase that stopped completing rather than any timing - Bound the ladder with a wall-time budget, a per-rung timeout, and a failure count, under the same lock and scratch-cleanup guarantees as the matrix supervisor - Render a Markdown table naming the surrendering phase per rung Ladders share a base shape, so axes stay comparable at equal pane pressure, and the tests pin that only one dimension moves per ladder.
why: The ladder crashed on its first rung. Shape defines __str__ but inherits object.__format__, which rejects any non-empty format spec, and the reporting loop pads shapes into a column. Unit tests and doctests all passed; only running the command found it. what: - Delegate Shape.__format__ to its string form - Pin column alignment in a regression, since that is the path the tests were missing
why: The notes still carried a two-sample cross-lane reading that twenty samples has since shown to be noise, and no record of how run sizes are chosen. what: - Record the four-cell result at twenty samples, including the strategy and ORM reference comparisons - Retract the earlier claim that the subprocess lanes won enumeration and search by 1.1x to 3.2x; those differences are unresolved - Record where each topology axis buckles and which phase surrenders, which differs by axis and would be lost by attributing a ceiling to panes alone - Explain how sample count and shape are chosen from measured variance rather than taste
why: The page showed how to compare lanes but not how to find where the workload stops completing, and said nothing about how to read a ratio the evidence cannot support. what: - Add a paste-and-run block for the pressure ladder, and say plainly that a rung's timing means nothing because only its outcome does - Explain that an unresolved row means the lanes are the same for that phase, not that a measurement is missing
why: A plain invocation ran four cells at 1,600 panes with fifteen samples, which takes about an hour. Subprocess cost per iteration grows superlinearly with pane count -- roughly 9 seconds at 800 panes against 49 at 1,600 -- so shape is the cheaper lever, and 800 panes at twenty samples was measured end to end at just under 24 minutes. Twenty samples is also the point where p90 and p95 become reportable at all, so the smaller shape buys statistics the larger one could not afford. what: - Default to 40x20x1 at twenty timed samples - Pin the defaults in a test that states the budget rationale, so raising either value cannot silently push a plain invocation past an hour - Show the default invocation first in the docs and the larger shape as the deliberate, slower alternative - Record that the TMUX-clearing prefix is defensive rather than required, since every script already strips it from the children it spawns
why: A supervisor run as a PEP 723 script gets an ephemeral environment with no libtmux, and spawns its children from it. The children then die on import inside a log nobody reads, so the run reports a phase failure that says nothing about the real cause. Removing the shebang does not fix this: uv run --script still works without the metadata, so the path stays reachable however it is reached. what: - Verify the child interpreter can import libtmux before spawning anything, and refuse with a message naming the project-environment invocation - Probe once per process rather than once per cell, since the answer cannot change while it runs - Cover both the refusal and the supported path, clearing the cache in the guard test so a stale entry cannot mask it
why: The benchmark requires Linux pidfd to bind exact process identity, and free-threaded CPython builds do not expose os.pidfd_open. uv prefers the interpreter it manages, so a fresh checkout can select a free-threaded build and refuse on a host where the benchmark ran yesterday. Refusing is correct: measuring without exact process identity would be worse than not measuring. Refusing with no way out is not. what: - Name the free-threaded cause, give the one-line check that confirms it, and point at UV_PYTHON as the way to select another interpreter - Cover the message, since a refusal a reader cannot act on is the defect Note that .python-version cannot express this: both a patch-level pin and an explicit non-freethreaded identifier still resolved to the managed free-threaded build, so the message points at the environment rather than pretending the repository can fix it.
why: The page called every script self-contained. Only plan is: it reads host files, imports no libtmux, and starts no server. run and ramp import libtmux from the working tree, so a reader following that claim reaches an environment where the benchmark cannot work. what: - State which commands are standalone and which need the project environment - Record that the benchmark requires an interpreter exposing pidfd, and what to do when a fresh checkout refuses for that reason - List the two supervisors alongside the scripts they drive - Clear VIRTUAL_ENV in the documented prefix, noting the whole prefix is defensive and that clearing it only silences a uv warning; the resolved interpreter is identical either way
why: The reasoning lived only in three stashes and a chat transcript. Dropping the stashes would take the decision record with them, leaving a guard whose rationale a future reader would have to reconstruct. what: - Record the two rejected strategies, what each actually did when built, and the premise under which each would have won instead - Note that the free-threaded interpreter finding only appears in a fresh checkout, which is why building the contenders surfaced it
why: The observer surface and the synchronous wrapper need nothing beyond the TmuxEngine protocol, so they live at the command execution seam and every engine gets them for free. An asynchronous engine protocol is experimental, which is the one part that cannot follow them down. Leaving a second copy here would recreate exactly the drift the seam exists to prevent: two definitions of what a command group costs, free to disagree. what: - Add AsyncInstrumentedEngine, and an instrument() that dispatches on the engine's own run() rather than on a flag - Re-export Sink, CountingSink, and InstrumentedEngine from the seam, so the experimental engines still present one surface - Take command_count from libtmux.engines.base everywhere, dropping the duplicate definition in control_mode
why: Tracing charges concurrent requests to separate spans, and that is the property that rules out installing observers on the engine or in an ambient scope: two overlapping scopes share one engine, so anything the engine holds cannot tell their commands apart. Nothing pinned it, so a future change could have moved the observer somewhere shared and stayed green. what: - Add a test running two interleaved scopes against one engine, asserting each sees only its own commands (a shared observer reports 5 and 5)
why: The counts only mean something read against a transport, and the zero-overhead property is a design consequence a reader has to be told, not something visible from the API surface. A reader reaching for call counts needs to know the seam does not extend there, and what to use instead. what: - Add docs/experimental/instrumentation.md covering counting, what the three counts mean per lane, writing a sink, exporting spans, async behavior, and the overhead argument - Drive an OpenTelemetry-shaped sink through a recording tracer so the span lifecycle is executed rather than described - Document counting Python calls around a whole run rather than per command: cProfile and sys.monitoring are process-global and single-owner, so a per-command counter would charge one command for work that overlapped it on the event loop - Pair the call count with the tmux command count, so a change that moves cost between them is visible as movement rather than as a win - Record why observation is neither ambient nor engine-installed: a consulted scope costs every call about eight percent forever, and an engine missing from its coverage list under-reports silently, because control mode splits dispatch between its connection and a subprocess fallback - Link it from the experimental index and the engines landing page
why: Each carries a `uv run --script` shebang and the docs invoke them directly, but they shipped without the executable bit, so the shebang described an invocation that could not work. ruff's EXE001 flags this on Linux; the rule is skipped on Windows and WSL, so it only surfaces in CI. what: - Record bench_orchestration, orchestration_fuzzer, orchestration_matrix, and orchestration_stress as mode 755, matching the other runnable scripts
why: CI type-checks the whole tree (`mypy .`), which includes `scripts/`;
running plain `mypy` locally uses the configured file list and skips them, so
a bare `t.cast("dict", ...)` here type-checked clean on a developer machine
and failed the gate. Six casts named `dict` with no arguments, which mypy
rejects under the project's strictness.
what:
- Spell the phase tables as `dict[str, t.Any]` at each cast site
…e inventory why: The inventory guard gives every concrete transport a reference page and excludes the wrappers that decorate them, keyed on the defining module so a new transport still fails it. The synchronous wrapper now lives at the command execution seam, so its defining module changed and the exclusion stopped matching -- InstrumentedEngine was counted as a transport owing a page it could not fill. what: - Key the exclusion on both instrumentation modules, the seam's and the experimental one, keeping the by-module rule rather than falling back to matching names
why: AF_UNIX bounds a bound path at 107 bytes plus its NUL. pytest's tmp_path already spends most of that on the user name and the test name, and under xdist it adds a per-worker segment as well, so a 25-character directory name put this one at exactly 108 bytes on CI and it failed on every tmux version. It passed locally only because the user name there is shorter -- five characters of difference decided it. what: - Shorten the directory to leave the budget fourteen bytes of headroom - Assert the encoded length before binding, so the constraint is checked rather than implied by a name nobody measures
why: Thirty cases in this file fail on tmux 3.2a while every other supported version passes them, so the 3.2a cell reported a wall of red that told a reader nothing they could act on. The cases that fail are the ones that build a real topology and drive it end to end; the rest of the file is unaffected, which is why the floor sits on those thirteen functions rather than on the module. The failures are mixed assertions, timeouts, and cancellations with no resource-exhaustion signature, and the control-mode engine itself passes on 3.2a one branch below. Nobody has diagnosed the cause, so this records the floor as unverified rather than claiming the harness is known-broken there. what: - Skip the live-topology and supervisor cases below tmux 3.3a, keeping the other 442 cases running on every supported version
why: The capture-timeout tests read the child's pid from a file the stub writes on startup, and the timeout under test is 50ms. The stub was a Python script, so it had to start an interpreter before recording anything -- longer than 50ms on a busy machine. The child was then killed before it wrote the file and the test failed reading it, which reads as a reaping fault when reaping had in fact worked. Measured at roughly one failure in twelve runs before the change and none in thirty after, across both the sync and async cases. what: - Write the stub as a shell script, so recording the pid costs milliseconds rather than interpreter startup - Use exec for the blocking sleep, so the recorded pid stays the process the test later asserts was reaped, and no second process outlives it
why: kill-server exits 1 when no server is listening, which is the state teardown exists to reach. A server can finish exiting on its own between the ownership check and the kill landing, and the larger the topology the more routinely it does -- an 80x20x1 rung loses that race as a matter of course. The exit code was treated as authoritative, so a teardown that had demonstrably worked (processes absent, socket absent) recorded an error; complete cleanup requires an empty error list, so the rung failed. The escalating stress harness stops on a failed rung, which reported the pane ceiling as the point where cleanup was misread rather than where anything actually broke. what: - Judge teardown on the owning process's absence, which the code already established, rather than on the helper's exit status - Keep reporting any other non-zero status, which is the helper behaving unexpectedly rather than tmux reporting an absent server - Say so when a server outlives a kill that reported no server, once its survival is established - Pin both directions: the old behaviour fails the new test, and dropping the anomalous-exit report fails the existing one
why: --_test-stall-after parks a worker mid-run so the cancellation tests can prove the supervisor reaps it, and the wait was `while True`. That is correct only while the test is alive to do the reaping. A test that fails, times out, or is interrupted leaves the worker parked forever -- and because preflight refuses to start while any benchmark process is running, one orphan blocks every subsequent benchmark on the machine until reboot. Found by being bitten: a worker stranded by a load-flaked run was still parked 1h44m later, its pytest temporary directory long deleted, holding a tmux server and a tail alongside it, and it refused three unrelated matrix runs before anyone connected the two. The tests driving that path set the progress watchdog at 0.2 and 0.3 seconds. That is a no-progress timer, so a test parking a worker at a checkpoint has to let the run reach it first, and the slowest step before any stall point is starting a tmux server -- well under a second idle, seconds on a loaded machine. The threshold is not what either test asserts; they check what the run recorded. Measured at load 19, one passed 2 of 6 runs. what: - Bound the wait by a deadline and by the spawner going away, since reparenting is the portable signal that nobody is left to cancel - Take the bound as an argument so it is testable without patching a global - Cover both exits, plus that the shipped bound is finite - Raise both watchdogs to a shared named bound, with the constraint stated once, and widen the cleanup grace to match
Two prose references still named `scripts/bench_engines.py`, which moved to `scripts/bench/engines.py` one branch below. Both are the kind of pointer a reader follows to decide whether a script is the one they want, so a stale path costs them the lookup.
The orchestration benchmark and its three drivers sat loose in `scripts/`
under a `bench_`/`orchestration_` prefix convention, beside `scripts/bench/`
and `scripts/lgtm/`, which are directories. Prefixes were doing a directory's
job: `matrix.py` calls itself a thin supervisor around the benchmark and
`stress.py` explains itself by contrast with `matrix.py`, so the four are one
subsystem and now read as one.
scripts/bench_orchestration.py -> scripts/orchestration/benchmark.py
scripts/orchestration_fuzzer.py -> scripts/orchestration/fuzzer.py
scripts/orchestration_matrix.py -> scripts/orchestration/matrix.py
scripts/orchestration_stress.py -> scripts/orchestration/stress.py
Sibling lookups (`Path(__file__).with_name(...)`) keep working because all
four moved together; only the names they ask for changed. Two things did not
survive mechanically:
- `--help` used to say `orchestration_stress.py`, which told a reader where to
find the script. Bare `stress.py` would not, so each parser now names its
path, and the doctests that pin `prog` moved with it. The benchmark's main
parser never set `prog` at all and had been relying on the filename, so it
gains one rather than regressing to `benchmark.py`.
- `matrix.py` finds running benchmarks by matching process lines against
filenames. `benchmark.py` and `fuzzer.py` are generic enough to match an
unrelated process, so the needles carry the directory now and are stricter
than before.
`tests/test_bench_orchestration_script.py` becomes
`tests/test_orchestration_benchmark.py`, matching its three siblings.
Verified: all four run from the new location, `--help` reports the new paths,
the documented `uv run --script ... plan` standalone path still works, and the
552 tests across the four modules pass.
The four cover `scripts/orchestration/`, so they sit beside it now and the pairing is readable from the path alone. Their repository-root lookups count directories, and the counts move with them. `fuzzer.py` pointed back at one of them by its old name.
tony
force-pushed
the
engine-ops-orchestration-bench
branch
from
August 23, 2026 18:09
5371eec to
45cfb97
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/bench_engines.py, which measures repeated construction of short-lived topologies.subprocess.Popenfrom outside — an approach that under-reports engines which never fork and cannot see command groups at all.Changes by area
Benchmark harness
scripts/bench_orchestration.py:plan,run, andrampcommands. Supervises its own worker process, binds socket ownership, publishes it atomically, drains repeated interrupts, and finalizes exactly once.scripts/orchestration_fuzzer.py: PEP 723 Rich workload generator withpreviewandserve, producing deterministic streams with recorded sentinel timing.scripts/orchestration_matrix.py: supervises lane comparisons one cell at a time and renders per-phase results, including a classic-ORM reference cell so the typed lanes have something to be measured against.scripts/orchestration_stress.py: escalating ladder that finds where each axis buckles.Engine observation
src/libtmux/experimental/engines/instrumentation.py:InstrumentedEngine,AsyncInstrumentedEngine, theSinkobserver surface, andCountingSink, reporting requests, tmux commands, commands inlined into another request's argv, and elapsed time.src/libtmux/experimental/engines/async_control_mode.py: await reconnect readiness rather than assuming it.Documentation
docs/experimental/orchestration-benchmark.md: purpose, deliverables, how to run, and the scale guide.docs/experimental/instrumentation.md: what the counts mean per lane, writing a sink, exporting spans, and the overhead argument.plans/orchestration-benchmark-scale.md: the measured comparisons and the decision record behind the interpreter check.Design decisions
Observation is a protocol decorator, not an engine feature or an ambient scope. Tracing charges concurrent requests to separate spans. Two overlapping scopes share one engine, so anything the engine holds cannot tell their commands apart. A consulted ambient scope costs every call about eight percent forever, and an engine missing from a coverage list under-reports silently, because control mode splits dispatch between its connection and a subprocess fallback. A program that wraps nothing constructs nothing and runs unchanged code.
Call counts are run-scoped, not per-command. cProfile and
sys.monitoringare process-global and single-owner, so a per-command counter would charge one command for work that merely overlapped it on the event loop.The benchmark refuses a free-threaded interpreter. Measuring without exact process identity would be worse than not measuring, and free-threaded CPython builds do not expose
os.pidfd_open. The refusal names the cause and points atUV_PYTHON, because.python-versioncannot express it — both a patch-level pin and an explicit non-freethreaded identifier still resolved to the managed free-threaded build.The default shape fits a sensible budget. Subprocess cost per iteration grows superlinearly with pane count — roughly 9 seconds at 800 panes against 49 at 1,600 — so shape is the cheaper lever than sample count. Twenty samples is also the point where p90 and p95 become reportable at all, so the smaller shape buys statistics the larger one could not afford.
Only
planis standalone.runandrampimport libtmux from the working tree. A supervisor run as a PEP 723 script gives its children an interpreter without libtmux, and they now refuse with an explanatory message rather than dying on import inside a log nobody reads.Test plan
uv run ruff check .— lint cleanuv run ruff format .— formatting clean, tree unchangeduv run mypy— types cleanuv run pytest --reruns 0— see the note below on load-sensitive testsjust build-docs— docs build cleantest_instrumentation.py— two interleaved scopes against one engine each see only their own commands; a shared observer would report both countstest_orchestration_matrix.py— warmup finishes before verify; the shipped defaults stay inside the stated budgettest_orchestration_fuzzer.py— workload determinism and delayed matchingtest_bench_orchestration_script.py— topology truth, phase correctness, cleanup, supervised recovery, and machine-readable resultstests/test_bench_orchestration_script.pycarries a stall/cancellation family that is load-sensitive above roughly load 20; those tests pass in isolation and their failing subset varies between runs on identical trees.