Repository navigation
test(auto-reply): run the whole runtime suite hermetically in CI and rewrite AGENTS.md - #53
Merged
Merged
Conversation
Every branch that edited the crate has landed, so the whole crate is formatted in one commit. No code changes; cargo test and clippy match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crate is now rustfmt-clean, so CI and release check it. AGENTS.md is rewritten to Anthropic's guidance for project instructions: short, only what the code cannot show, check commands first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ripting Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g why it failed The GeekNews room readback asks local-read for 100 rows but capped stdout at the 64 KiB model-output limit. Pretty-printed rows with message text and raw attachment JSON exceed that, so the baseline failed as 'unable to establish room readback baseline' depending on room content. The readback now has a 4 MiB cap, and every failure prints one reason code (binary_unavailable, timeout, output_over_cap, exit_N, ...) to stderr without message content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_scheduled_skip_paths_use_strict_lost_ack_projection built its scheduled job without the analysis watermark and without a current room, so every case was deferred as context_freshness_unavailable. d4b2ca7 made a scheduled send require a stable room watermark equal to the job's analysis watermark. The job now records one, as production does at schedule time. Two cases asserted paths the worker no longer has: - "context" patched is_context_only_author, which the scheduled path never reads (not even in 2672654). A persona-authored job is skipped through the numeric identity hold, so the case now returns identity "self" and expects self_author. - "short-laughter" expected the fixed reply_laughter_policy_violation reason. a54c3c3 records the first policy rule a draft breaks, and a two-ㅋ draft breaks reaction_mismatch first, so it expects policy_reaction_mismatch. The assertion is unchanged: a lost context ack is projected as skipped only when the readback matches the exact terminal decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three recovery tests insert a delivery_unknown job and record its AX attempt only as pipeline_transitions rows. 16d4c12 moved that fact onto the job row (reply_jobs.ax_mutation_authorized, set by the status trigger when a job enters sending) and 9d3f317 made leftover_unknown_has_ax_mutation read it, so these rows looked like jobs that never pressed Return. A queue v3 job cannot reach delivery_unknown after an AX attempt without that flag. The fixtures now insert it, and every recovery assertion is unchanged: - test_high_attempt_ax_authorized_unknown_skips_without_retransmit - test_ax_confirmed_unknown_heals_to_sent_without_retransmit - test_ax_authorized_unknown_without_confirm_stays_delivery_unknown Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two tests were written before same-author supersession became an edge in reply_job_supersessions, and before a successor carried its in-burst predecessors forward. test_supersession_race_linearizes_before_sending: - 278b3a6 links a pending, processing or scheduled predecessor when its successor is enqueued (pinned by test_newer_same_author_supersedes_inflight_ generation). A successor that commits before the predecessor's sending CAS therefore wins it: the CAS returns "superseded" and the row becomes projection_pending/burst_superseded without send authority. - Once sending has committed, no edge is recorded and the predecessor stays sending. 06955d0 carries the predecessor's row forward in the successor's burst IDs, so the successor is queued with [401, 402], not the singleton [402] that the 2672654 burst rule produced. test_burst_projection_retries_audit_without_delivery_authority expected the first claim to report projection_pending. Enqueue records the edge and leaves the predecessor pending; processing then projects it. The test now asserts the edge and the pending claim, and the retry and audit assertions are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_questions_and_advice_use_only_the_immediate_component still expected a social reply to be forced onto the short component and clamped to 20 s. That was the pacing before b8d80f1 ("sample the learned pacing mixture and charge only the remaining wait") and e77399b ("pace by turn kind"). CHANGELOG records the change: the old rule forced questions to immediate and everything else to short with a cap, so the learned mixture never applied. The question and advice assertions are unchanged. The two tail cases now assert the current contract: a statement keeps the learned mixture with no scheduled cap (40 s stays 40 s), and a caption prefers the short component (140 s stays 140 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_recent_tail_persists_across_polls_in_log_order checked that the bounded tail has RECENT_MESSAGE_LIMIT rows and then hard-coded its first log ID as 7. Out of rows 1 to 19 that is only true for a 13-row window. db-watch and the worker both set RECENT_MESSAGE_LIMIT = 7 (since 2672654), and the worker reads only raw[-RECENT_MESSAGE_LIMIT:], so the tail keeps rows 13 to 19. The test now asserts the whole kept window in log order instead of one restated boundary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_stale_backlog_uses_sample_distribution_upper_before_model used "old question?" as its expired inbound and expected a stale_backlog skip. analyze_event exempts a real question from that gate, so the event reached the model. The exemption is a deliberate rule applied across the worker (both analyze paths and three recovery paths), and CHANGELOG records that real questions keep reply authority. The stale case now uses a statement of the same age and still asserts the skip happens before the model. A second case pins the exemption: the same question past the window reaches the model exactly once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ran test_post_send_failure_with_successor_stays_delivery_unknown counted calls to conversation_advanced_past_event and expected two. d4b2ca7 replaced the pre-send freshness check with a stable room watermark that must equal the job's analysis watermark, so that helper now runs once. The count pinned an internal call order, not anything a user of the queue can see. The watermark gate itself is pinned in tests.test_auto_reply_turn_holds and tests.test_auto_reply_send_path. The room is current through the shared fixture, and the test asserts the outcome instead: exactly one send attempt, the job stays delivery_unknown with its formed reply, and no skip audit is written for the successor race. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pens test_reply_worker_reopens_after_transient_sqlite_busy made recover_stale_jobs raise "database is locked" and expected the worker to fence, sleep and reopen the queue. d52c96e deliberately keeps that one error inside the loop, so the reconcile release pass still runs under sustained SQLITE_BUSY. Its own test, test_release_runs_even_when_recovery_raises_transient_busy, pins that. The worker then reached the release pass with the test's Mock connection and exited 1. The reopen path itself is unchanged: a busy error from any other queue operation drops the connection, whose last commit outcome is unknown, fences sqlite_busy, sleeps once and reopens. The test now raises the busy from claim_job and keeps every original assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_fresh_media_unavailable_clarification_uses_standard_scheduler expected a
fresh unreadable photo to schedule MEDIA_UNAVAILABLE_CLARIFICATION
("사진이 안 열리네 다시 보내봐") as a question. The worker schedules a
_photo_fallback_reply line as a social reply instead, and no production code
reads MEDIA_UNAVAILABLE_CLARIFICATION any more. The reply-decision prompt in the
same file says "Never send a stock photo-ack such as 이거 뭐야 / 사진 보냈네 /
사진이 안 열리네", so the test held the opposite of a recorded product rule.
The scheduling assertions are unchanged: scheduled on the standard 5 s timing,
reason media_unavailable_clarification, no model call, no send. The reply is
checked as non-empty and free of the forbidden ack rather than as one exact
line, because _photo_fallback_reply seeds its choice from Python's per-process
string hash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bundle Two tests expected finish_delivery_unknown and the author terminal skip to delete the job's owned media bundle. Both call release_media_event, whose contract is "Scrub durable path capabilities without deleting a failed analysis bundle". The bundle is then reclaimed by db-watch's orphan sweep after MEDIA_ORPHAN_TTL_SECONDS (24 h + 65 s). The function and the tests both date from 2672654, so history cannot say which side is newer, and the code's contract is the one written down. Pin the documented behavior under names that say it: the durable event loses every path capability, the job reaches its terminal status, and the bundle directory still exists. The author case now drives finish_turn_policy_skip, the production entry point, instead of finish_author_identity_policy_skip, which nothing in scripts/ calls. Whether a terminal no-send path should delete the bundle at once is an open product question for the owner: after the scrub nothing can reach the bundle, so it only waits for the 24 h sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e48e04e made product generation local-only. _active_reply_model replaces any cloud REPLY_MODEL with Flash-Next, and _run_generation_candidate sends an MLX-Serve model to the gateway on 127.0.0.1:11234. So these tests, which stubbed the codex process, reached the live model port, and the hermetic guard refused it: - test_generate_reply_always_bypasses_runner_hash_cache - test_direct_target_prompt_is_strong_but_allows_low_value_skip - test_direct_target_reply_requires_exact_target_evidence - test_codex_rate_limit_circuit_persists_without_sensitive_output The module already had _fake_qwen27b_gateway, which replaces auto_reply_ondevice._local_only_urlopen, the only way the worker reaches the gateway. It becomes _fake_mlx_gateway: it advertises the requested local model and can answer the chat endpoint with an HTTP error. Its three image callers pass the 27B id. _load_local_mlx_module loads the worker on the profile that validate_auto_reply_local_mlx_profile accepts (opencodex runner, Flash-Next). - The two direct-target tests run on that profile. They read the prompt from the gateway's user message instead of the codex stdin framing. - The rate-limit test (renamed test_model_rate_limit_circuit_persists_ without_sensitive_output) runs on that profile too. The gateway answers 429 with a Retry-After hint. It asserts the same durable open circuit, the refusal after a module restart with no second model call, and the absence of every private marker from the queue file. - The hash-cache test keeps the codex configuration, because only a codex runner is rehashed, and now also checks that each call reached the gateway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runner test_codex_luna_max_fast_runner_is_pinned_and_parses_jsonl asserted that generate_reply runs "codex exec" with gpt-5.6-luna, max effort and the priority tier, and parses its JSONL. Since e48e04e that cannot happen. README says the production worker "permit[s] only local MLX for text and image generation ... Gemini and other cloud models are rejected as primary or fallback models", and _active_reply_model swaps any cloud id for Flash-Next. The codex command is still built but never spawned. The test is now test_codex_luna_configuration_generates_only_on_local_flash_next. With the same trusted Luna configuration it asserts the current contract: no runner process and no non-local HTTP, exactly one gateway request for Flash-Next that carries the private message and context, the reply rules in that request, and the gateway's JSON decision parsed into the result. Open product question for the owner: src/config.rs still accepts reply_runner_kind = "codex" with gpt-5.6-luna (and gjc with Gemini) under remote_explicit, so an operator can configure a cloud runner that the worker silently replaces with local Flash-Next. Either startup should reject it, or the codex command builder, its JSONL unwrapping and the Luna capacity probe are dead code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests.test_auto_reply_turn_holds borrows six helpers from tests.test_auto_reply_cli_runtime and builds one AutoReplyCliRuntimeTests instance for _load_trusted_codex_module, but it never started the harness. Its tests ran with the caller's HOME and OPENKAKAO_* environment and with no socket or file guards, so a run from a normal shell could read the operator's real account state. Install the guards per module and give every test its own empty HOME, clean environment and violation check, exactly as the runtime module does. All 38 tests pass unchanged. A probe inside one of its tests confirmed that opening ~/.config/openkakao/config.toml and connecting to 127.0.0.1:11234 are both refused. AGENTS.md now names both guarded modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it Five AX script tests called importlib.reload(auto_reply_ax_ui) so the module would re-read OPENKAKAO_TARGET_CHAT_NAME at import, then reloaded it again to "restore" it. A reload re-executes the shared module in place and mints new BackgroundAxResult and VirtualCursor classes. jarvis_tool_runtime had already imported the old ones, so after these tests its isinstance check rejected every AX outcome and reported FAILED instead of REFUSED or ABORTED. Running tests.test_auto_reply_cli_runtime before tests.test_jarvis_tool_runtime in one process failed two jarvis tests (test_focus_or_real_pointer_requests_are_refused_without_action, test_global_abort_during_each_job_overrides_success). Each passes alone. _load_ax_ui_module executes a private copy under a unique name and never touches sys.modules["auto_reply_ax_ui"]. The name is registered only while the module executes, because dataclasses resolves its string annotations through sys.modules. The five tests keep every assertion, and the pair now passes in one process (356 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The focused list ran 3 of the 348 tests in tests.test_auto_reply_cli_runtime, because the rest needed a real decision store, the live model server, or a fixture that had drifted from the queue's room layout. The module is now hermetic (tests/auto_reply_hermetic.py) and every test in it passes, so the three GeekNews ids give way to the whole module and the comment that explained the gap goes. Verified locally with Python 3.11 under HOME set to an empty temp dir and the guard sandbox profile: the parsed workflow list ran 1362 tests, OK, 9 skipped. The same list on stability-wave1 (9bac86b) ran 1017. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…os-15 runner label
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
twoimo
changed the base branch from
stability/4-send-once-and-geeknews
to
main
November 16, 2026 00:54
The hook launchers refuse an interpreter that group or others can write, as they must in production. The runner's toolcache Python fails that check, so five hook tests error with 'hook interpreter unsafe' in CI while they pass on a machine whose Python is owner-only writable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner's Python is root:admin with mode 775, so the plain chmod was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the CI runner the busy idle read took 3.16 s against a 2.5 s bound (1.5 s idle timeout + 1 s); locally it takes 1.55 to 1.6 s. SQLite's busy handler sums its planned short sleeps without checking real time, so each sleep running long on a loaded VM stretches the wait (plausible, not confirmed). The test exists to prove the idle read does not use the default 5 s wait: with that regression it takes 5.25 s and still fails the elapsed < 4.0 check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The auto-reply runtime test module sent real prompts to the owner's local model server and wrote a live state file, so CI could only run 3 of its 348 tests. AGENTS.md also had a broken code fence, a nonexistent flag, and advice that contradicted its own safety model.
What changed
HOMEwith socket and path guards; 16 stale or fixture-starved tests are fixed without production changes, and CI runs the whole module.rustfmt-clean and CI and release check it. GeekNews room readback gets its own 4 MiB cap and logs a reason code when it fails.Scope
Blast Radius
Tests and docs, a pure format commit, and one GeekNews readback change that only accepts larger valid pages.
Verification
cargo fmt --checkclean,cargo test1,168 passed.🤖 Generated with Claude Code