Skip to content

test(auto-reply): run the whole runtime suite hermetically in CI and rewrite AGENTS.md - #53

Merged
twoimo merged 24 commits into
mainfrom
stability/5-hermetic-tests-fmt-agents-md
Nov 16, 2026
Merged

twoimo merged 24 commits into
mainfrom
stability/5-hermetic-tests-fmt-agents-md

Conversation

@twoimo

@twoimo twoimo commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Why

The auto-reply runtime test module sent real prompts to the owner's local model server and wrote a live state file, so CI could only run 3 of its 348 tests. AGENTS.md also had a broken code fence, a nonexistent flag, and advice that contradicted its own safety model.

What changed

  • All 348 runtime tests and the turn-hold tests run under a per-test empty HOME with socket and path guards; 16 stale or fixture-starved tests are fixed without production changes, and CI runs the whole module.
  • The CLI crate is rustfmt-clean and CI and release check it. GeekNews room readback gets its own 4 MiB cap and logs a reason code when it fails.
  • AGENTS.md is rewritten per Anthropic's guidance for project instructions: under 200 lines, check commands first, only what the code cannot show.

Scope

  • Four product questions found by the test triage stay open (cloud runners, photo retention, photo fallback lines, burst race).

Blast Radius

Tests and docs, a pure format commit, and one GeekNews readback change that only accepts larger valid pages.

Verification

  • Runtime module 348/348, five runs, no flakes; CI Python list 1,362 passed under the sandbox guard.
  • cargo fmt --check clean, cargo test 1,168 passed.

🤖 Generated with Claude Code

twoimo and others added 21 commits November 16, 2026 04:42
Every branch that edited the crate has landed, so the whole crate is
formatted in one commit. No code changes; cargo test and clippy match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crate is now rustfmt-clean, so CI and release check it. AGENTS.md is
rewritten to Anthropic's guidance for project instructions: short, only
what the code cannot show, check commands first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ripting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g why it failed

The GeekNews room readback asks local-read for 100 rows but capped stdout
at the 64 KiB model-output limit. Pretty-printed rows with message text
and raw attachment JSON exceed that, so the baseline failed as
'unable to establish room readback baseline' depending on room content.
The readback now has a 4 MiB cap, and every failure prints one reason code
(binary_unavailable, timeout, output_over_cap, exit_N, ...) to stderr
without message content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_scheduled_skip_paths_use_strict_lost_ack_projection built its scheduled
job without the analysis watermark and without a current room, so every case
was deferred as context_freshness_unavailable. d4b2ca7 made a scheduled send
require a stable room watermark equal to the job's analysis watermark. The job
now records one, as production does at schedule time.

Two cases asserted paths the worker no longer has:

- "context" patched is_context_only_author, which the scheduled path never
  reads (not even in 2672654). A persona-authored job is skipped through the
  numeric identity hold, so the case now returns identity "self" and expects
  self_author.
- "short-laughter" expected the fixed reply_laughter_policy_violation reason.
  a54c3c3 records the first policy rule a draft breaks, and a two-ㅋ draft
  breaks reaction_mismatch first, so it expects policy_reaction_mismatch.

The assertion is unchanged: a lost context ack is projected as skipped only
when the readback matches the exact terminal decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three recovery tests insert a delivery_unknown job and record its AX attempt
only as pipeline_transitions rows. 16d4c12 moved that fact onto the job row
(reply_jobs.ax_mutation_authorized, set by the status trigger when a job
enters sending) and 9d3f317 made leftover_unknown_has_ax_mutation read it, so
these rows looked like jobs that never pressed Return.

A queue v3 job cannot reach delivery_unknown after an AX attempt without that
flag. The fixtures now insert it, and every recovery assertion is unchanged:

- test_high_attempt_ax_authorized_unknown_skips_without_retransmit
- test_ax_confirmed_unknown_heals_to_sent_without_retransmit
- test_ax_authorized_unknown_without_confirm_stays_delivery_unknown

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two tests were written before same-author supersession became an edge in
reply_job_supersessions, and before a successor carried its in-burst
predecessors forward.

test_supersession_race_linearizes_before_sending:
- 278b3a6 links a pending, processing or scheduled predecessor when its
  successor is enqueued (pinned by test_newer_same_author_supersedes_inflight_
  generation). A successor that commits before the predecessor's sending CAS
  therefore wins it: the CAS returns "superseded" and the row becomes
  projection_pending/burst_superseded without send authority.
- Once sending has committed, no edge is recorded and the predecessor stays
  sending. 06955d0 carries the predecessor's row forward in the successor's
  burst IDs, so the successor is queued with [401, 402], not the singleton
  [402] that the 2672654 burst rule produced.

test_burst_projection_retries_audit_without_delivery_authority expected the
first claim to report projection_pending. Enqueue records the edge and leaves
the predecessor pending; processing then projects it. The test now asserts the
edge and the pending claim, and the retry and audit assertions are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_questions_and_advice_use_only_the_immediate_component still expected a
social reply to be forced onto the short component and clamped to 20 s. That
was the pacing before b8d80f1 ("sample the learned pacing mixture and charge
only the remaining wait") and e77399b ("pace by turn kind"). CHANGELOG records
the change: the old rule forced questions to immediate and everything else to
short with a cap, so the learned mixture never applied.

The question and advice assertions are unchanged. The two tail cases now
assert the current contract: a statement keeps the learned mixture with no
scheduled cap (40 s stays 40 s), and a caption prefers the short component
(140 s stays 140 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_recent_tail_persists_across_polls_in_log_order checked that the bounded
tail has RECENT_MESSAGE_LIMIT rows and then hard-coded its first log ID as 7.
Out of rows 1 to 19 that is only true for a 13-row window. db-watch and the
worker both set RECENT_MESSAGE_LIMIT = 7 (since 2672654), and the worker reads
only raw[-RECENT_MESSAGE_LIMIT:], so the tail keeps rows 13 to 19.

The test now asserts the whole kept window in log order instead of one
restated boundary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_stale_backlog_uses_sample_distribution_upper_before_model used
"old question?" as its expired inbound and expected a stale_backlog skip.
analyze_event exempts a real question from that gate, so the event reached
the model. The exemption is a deliberate rule applied across the worker
(both analyze paths and three recovery paths), and CHANGELOG records that
real questions keep reply authority.

The stale case now uses a statement of the same age and still asserts the
skip happens before the model. A second case pins the exemption: the same
question past the window reaches the model exactly once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ran

test_post_send_failure_with_successor_stays_delivery_unknown counted calls to
conversation_advanced_past_event and expected two. d4b2ca7 replaced the
pre-send freshness check with a stable room watermark that must equal the
job's analysis watermark, so that helper now runs once. The count pinned an
internal call order, not anything a user of the queue can see. The watermark
gate itself is pinned in tests.test_auto_reply_turn_holds and
tests.test_auto_reply_send_path.

The room is current through the shared fixture, and the test asserts the
outcome instead: exactly one send attempt, the job stays delivery_unknown
with its formed reply, and no skip audit is written for the successor race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pens

test_reply_worker_reopens_after_transient_sqlite_busy made recover_stale_jobs
raise "database is locked" and expected the worker to fence, sleep and reopen
the queue. d52c96e deliberately keeps that one error inside the loop, so the
reconcile release pass still runs under sustained SQLITE_BUSY. Its own test,
test_release_runs_even_when_recovery_raises_transient_busy, pins that. The
worker then reached the release pass with the test's Mock connection and
exited 1.

The reopen path itself is unchanged: a busy error from any other queue
operation drops the connection, whose last commit outcome is unknown, fences
sqlite_busy, sleeps once and reopens. The test now raises the busy from
claim_job and keeps every original assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_fresh_media_unavailable_clarification_uses_standard_scheduler expected a
fresh unreadable photo to schedule MEDIA_UNAVAILABLE_CLARIFICATION
("사진이 안 열리네 다시 보내봐") as a question. The worker schedules a
_photo_fallback_reply line as a social reply instead, and no production code
reads MEDIA_UNAVAILABLE_CLARIFICATION any more. The reply-decision prompt in the
same file says "Never send a stock photo-ack such as 이거 뭐야 / 사진 보냈네 /
사진이 안 열리네", so the test held the opposite of a recorded product rule.

The scheduling assertions are unchanged: scheduled on the standard 5 s timing,
reason media_unavailable_clarification, no model call, no send. The reply is
checked as non-empty and free of the forbidden ack rather than as one exact
line, because _photo_fallback_reply seeds its choice from Python's per-process
string hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bundle

Two tests expected finish_delivery_unknown and the author terminal skip to
delete the job's owned media bundle. Both call release_media_event, whose
contract is "Scrub durable path capabilities without deleting a failed
analysis bundle". The bundle is then reclaimed by db-watch's orphan sweep
after MEDIA_ORPHAN_TTL_SECONDS (24 h + 65 s). The function and the tests
both date from 2672654, so history cannot say which side is newer, and the
code's contract is the one written down.

Pin the documented behavior under names that say it: the durable event loses
every path capability, the job reaches its terminal status, and the bundle
directory still exists. The author case now drives finish_turn_policy_skip,
the production entry point, instead of finish_author_identity_policy_skip,
which nothing in scripts/ calls.

Whether a terminal no-send path should delete the bundle at once is an open
product question for the owner: after the scrub nothing can reach the bundle,
so it only waits for the 24 h sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e48e04e made product generation local-only. _active_reply_model replaces any
cloud REPLY_MODEL with Flash-Next, and _run_generation_candidate sends an
MLX-Serve model to the gateway on 127.0.0.1:11234. So these tests, which
stubbed the codex process, reached the live model port, and the hermetic
guard refused it:

- test_generate_reply_always_bypasses_runner_hash_cache
- test_direct_target_prompt_is_strong_but_allows_low_value_skip
- test_direct_target_reply_requires_exact_target_evidence
- test_codex_rate_limit_circuit_persists_without_sensitive_output

The module already had _fake_qwen27b_gateway, which replaces
auto_reply_ondevice._local_only_urlopen, the only way the worker reaches the
gateway. It becomes _fake_mlx_gateway: it advertises the requested local
model and can answer the chat endpoint with an HTTP error. Its three image
callers pass the 27B id.

_load_local_mlx_module loads the worker on the profile that
validate_auto_reply_local_mlx_profile accepts (opencodex runner, Flash-Next).

- The two direct-target tests run on that profile. They read the prompt from
  the gateway's user message instead of the codex stdin framing.
- The rate-limit test (renamed test_model_rate_limit_circuit_persists_
  without_sensitive_output) runs on that profile too. The gateway answers 429
  with a Retry-After hint. It asserts the same durable open circuit, the
  refusal after a module restart with no second model call, and the absence
  of every private marker from the queue file.
- The hash-cache test keeps the codex configuration, because only a codex
  runner is rehashed, and now also checks that each call reached the gateway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runner

test_codex_luna_max_fast_runner_is_pinned_and_parses_jsonl asserted that
generate_reply runs "codex exec" with gpt-5.6-luna, max effort and the
priority tier, and parses its JSONL. Since e48e04e that cannot happen. README
says the production worker "permit[s] only local MLX for text and image
generation ... Gemini and other cloud models are rejected as primary or
fallback models", and _active_reply_model swaps any cloud id for Flash-Next.
The codex command is still built but never spawned.

The test is now test_codex_luna_configuration_generates_only_on_local_flash_next.
With the same trusted Luna configuration it asserts the current contract:
no runner process and no non-local HTTP, exactly one gateway request for
Flash-Next that carries the private message and context, the reply rules in
that request, and the gateway's JSON decision parsed into the result.

Open product question for the owner: src/config.rs still accepts
reply_runner_kind = "codex" with gpt-5.6-luna (and gjc with Gemini) under
remote_explicit, so an operator can configure a cloud runner that the worker
silently replaces with local Flash-Next. Either startup should reject it, or
the codex command builder, its JSONL unwrapping and the Luna capacity probe
are dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests.test_auto_reply_turn_holds borrows six helpers from
tests.test_auto_reply_cli_runtime and builds one AutoReplyCliRuntimeTests
instance for _load_trusted_codex_module, but it never started the harness. Its
tests ran with the caller's HOME and OPENKAKAO_* environment and with no
socket or file guards, so a run from a normal shell could read the
operator's real account state.

Install the guards per module and give every test its own empty HOME, clean
environment and violation check, exactly as the runtime module does. All 38
tests pass unchanged. A probe inside one of its tests confirmed that opening
~/.config/openkakao/config.toml and connecting to 127.0.0.1:11234 are both
refused. AGENTS.md now names both guarded modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it

Five AX script tests called importlib.reload(auto_reply_ax_ui) so the module
would re-read OPENKAKAO_TARGET_CHAT_NAME at import, then reloaded it again to
"restore" it. A reload re-executes the shared module in place and mints new
BackgroundAxResult and VirtualCursor classes. jarvis_tool_runtime had already
imported the old ones, so after these tests its isinstance check rejected
every AX outcome and reported FAILED instead of REFUSED or ABORTED.

Running tests.test_auto_reply_cli_runtime before tests.test_jarvis_tool_runtime
in one process failed two jarvis tests
(test_focus_or_real_pointer_requests_are_refused_without_action,
test_global_abort_during_each_job_overrides_success). Each passes alone.

_load_ax_ui_module executes a private copy under a unique name and never
touches sys.modules["auto_reply_ax_ui"]. The name is registered only while
the module executes, because dataclasses resolves its string annotations
through sys.modules. The five tests keep every assertion, and the pair now
passes in one process (356 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The focused list ran 3 of the 348 tests in tests.test_auto_reply_cli_runtime,
because the rest needed a real decision store, the live model server, or a
fixture that had drifted from the queue's room layout. The module is now
hermetic (tests/auto_reply_hermetic.py) and every test in it passes, so the
three GeekNews ids give way to the whole module and the comment that
explained the gap goes.

Verified locally with Python 3.11 under HOME set to an empty temp dir and
the guard sandbox profile: the parsed workflow list ran 1362 tests, OK,
9 skipped. The same list on stability-wave1 (9bac86b) ran 1017.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@twoimo
twoimo changed the base branch from stability/4-send-once-and-geeknews to main November 16, 2026 00:54
@twoimo twoimo closed this Nov 16, 2026
@twoimo twoimo reopened this Nov 16, 2026
twoimo and others added 3 commits November 16, 2026 10:05
The hook launchers refuse an interpreter that group or others can write,
as they must in production. The runner's toolcache Python fails that
check, so five hook tests error with 'hook interpreter unsafe' in CI
while they pass on a machine whose Python is owner-only writable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner's Python is root:admin with mode 775, so the plain chmod was
refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the CI runner the busy idle read took 3.16 s against a 2.5 s bound
(1.5 s idle timeout + 1 s); locally it takes 1.55 to 1.6 s. SQLite's busy
handler sums its planned short sleeps without checking real time, so each
sleep running long on a loaded VM stretches the wait (plausible, not
confirmed). The test exists to prove the idle read does not use the
default 5 s wait: with that regression it takes 5.25 s and still fails the
elapsed < 4.0 check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@twoimo twoimo closed this Nov 16, 2026
@twoimo twoimo reopened this Nov 16, 2026
@twoimo
twoimo merged commit cc25f40 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant