Skip to content

refactor(auto-reply): pin and simplify the reply worker and fix dormant relay bugs - #51

Merged
twoimo merged 27 commits into
mainfrom
stability/3-worker-refactor-and-dormant-modules
Nov 16, 2026
Merged

twoimo merged 27 commits into
mainfrom
stability/3-worker-refactor-and-dormant-modules

Conversation

@twoimo

@twoimo twoimo commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Why

scripts/auto-reply-worker.py (17,633 lines) composes and sends every automatic reply, and the dormant Rust relay, forward and collector modules had confirmed bugs that would resend messages once wired in.

What changed

  • Worker refactor without behavior change: 52 send-path pin tests first, then dead code removed, one skip finisher, one fallback-model path, process_job split into its two phases (17,633 → 17,332 lines).
  • Relay retries only missing images within 30 s, forwarder records >10-image posts, store errors fail closed, link keys keep path case, quality drop of exactly 5 rolls back (spec R8.12).
  • A failed relay send now holds that room's queue until a send succeeds, and a relay message over 10 images commits once the first 10 arrive. The collector enforces its call bounds, reports a failed health check as a failure, caps a fallback page at 1 MiB, and fails a collection whose temp data it cannot delete.
  • CI runs the new send-path, metrics and supervisor-lifecycle tests.

Scope

  • The dormant modules stay unwired; only their bugs are fixed.

Blast Radius

The worker refactor is pinned test by test at every commit, plus a model check that one reply job posts at most once. The dormant modules have no production caller.

Verification

  • Each worker commit: 953 tests identical to base; the send-once model check unchanged.
  • cargo test 1,164 passed, clippy clean.

🤖 Generated with Claude Code

twoimo and others added 26 commits November 16, 2026 02:01
tests/test_auto_reply_cli_runtime.py is not hermetic and most of it does not
run in CI. Its tests for the scheduled send phase fail on a clean machine, so
nothing pinned what the worker does between claiming a due draft and recording
the outcome.

Add 50 tests that load the worker with a private home, queue, and decision
store. They replace the one process seam with a fake openkakao-cli that keeps
a decision store and a room transcript. A guard fails any test that starts a
process or opens a socket.

The tests pin four areas:

- Delivery. A draft is recorded as sent only after the local database shows
  the reply. An unconfirmed send becomes delivery_unknown and is never retried.
- Skips. Each site that closes a job without a send writes its audit first and
  keeps its own queue projection.
- The send boundary. `send_reply` refuses a policy violation, an author
  outside the allowlist, and a closed fence before any command. It never
  composes for a job that is already sending.
- Fallbacks. `generate_reply` keeps the lease of the model that answered at
  each of the four places it consults the fallback chain.

A mutation check with 73 single-edit mutants of the worker kills 70. The other
three cannot change behavior: a boolean log id, a branch that repeats the
branch above it, and a fallback candidate that its own lease already refuses.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A test module that CI does not run goes stale. In a clean home, 73 of the 348
tests in tests/test_auto_reply_cli_runtime.py fail or error, and CI runs only
three of them.

Add tests.test_auto_reply_send_path to the focused list. It needs no network,
no process, and no state outside its own temporary directory.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Ten names in scripts/auto-reply-worker.py have no reference anywhere in the
repository, and two constants are defined twice with the same value. A reader
could not tell that from the file and had to trace each one.

Delete them:

- is_reply_author, _immediate_burst_predecessor, _predecessor_event_matches,
  processing_job_has_no_send_attempt, _style_topic_candidates,
  _mark_geeknews_slot_posted, and _proactive_search_hits
- REPLY_MEMORY_LIMIT, PROACTIVE_EVENT_PREFIX, and PROACTIVE_TOKEN_DENYLIST
- the second definitions of MEDIA_DIR_PREFIX and MEDIA_ACTIVE_MARKER

PROACTIVE_TOKEN_DENYLIST was used only by _style_topic_candidates. The worker
has no dynamic name lookup, so a name with no textual reference is unreachable.

The deletion is 174 lines. A script removed the definitions and then checked
that the other 664 top-level statements parse to the same syntax trees as
before.

Five more names are used only by tests. They stay, because removing them
would change tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e outcome

Three kinds of condition on the send path could never decide anything. Each
one made a reader stop and work out what it guarded.

- `send_reply` returns early unless the CLI result is a dict, then tested
  `isinstance(result, dict)` four more times. Remove the four later tests.
- `send_reply` required `str(...) is not None` before it looked for the reply
  in the local database. `str()` never returns None. Remove the test.
- `process_job` handled a job left in `sending` with the same call as the
  branch two below it, which handles every phase but `processing`. The branch
  between them tests for `projection_pending`, which a `sending` job is not.
  Remove the first branch.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e connection

`process_job` accepts a missing queue connection. In that case
`settle_processing_transition` writes the queue row without checking its
status. A skip whose audit failed must still end as delivery_unknown, not as
skipped.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Eight places closed a job that sent nothing with the same three steps: write
the no-send audit, move the queue row to `skipped`, and complete the event.
Each place spelled the steps out again, about 30 lines at a time, and the
copies had drifted. One keeps the draft on the skipped row. Two project
nothing onto the row when the audit fails. A reader had to diff the copies to
find that out.

Add `finish_proven_skip`, which owns the three steps and their early returns.
Each caller now passes only what it does differently:

- `audited_event` when the audit describes the whole burst
- `rewrite_event` when the stored event is replaced by the scrubbed one
- `unknown_fields` when a failed audit should not project the skip
- the remaining queue columns

`finish_conversation_advanced`, `finish_scheduled_stale_backlog`,
`finish_turn_policy_skip`, and five sites in `process_job` call it.
`finish_burst_superseded` and the hook in `main` follow other rules and stay
as they are.

Every caller makes the same calls with the same arguments as before, in the
same order. The worker shrinks by 90 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
When the generation budget is already spent, `generate_reply` must not call
the primary model, and the fallback chain must not open a lease for any
candidate. The retry-policy tests cover the chain alone. This test covers the
chain as `generate_reply` calls it, for both runner kinds, with and without a
cooled-down primary.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`generate_reply` asked the fallback chain for an answer in five places. Each
place rebuilt the same candidate runner. Three of them then repeated the same
six assignments to move the turn onto the model that answered. A fix in one
copy has missed the others before, which is why
tests/test_auto_reply_fallback_lease.py counts the copies.

Give the function one of each:

- `fallback_winner` builds the candidate runner and calls the chain.
- `continue_on_fallback` closes the failed attempt's lease and moves the turn
  onto the winner's lease, model, and output.

The cooldown branch had one arm per runner kind with identical bodies. It is
now one call. The primary-limit, timeout, and quota branches call both
helpers. Each branch passes the same arguments to the chain and closes the
failed lease with the same failure class and retry hint as before.

Update the two counts in test_one_shared_chain_leases_each_candidate. They
now require exactly one chain call and exactly one lease hand-over in
`generate_reply`, so a second copy fails the test.

The function shrinks by 78 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`process_job` ran two different jobs in one 660-line body. A job claimed from
`scheduled` carries a draft to send. Any other claimed job has to be analyzed
first. The first job filled a 320-line `if` block that every path left with
`return`, and the second job followed it. A reader who wanted one phase had to
scroll past the other and track which `return` belonged to which.

Move each phase into a function:

- `_deliver_scheduled_reply` is the old `if previous_status == "scheduled"`
  block.
- `_analyze_and_schedule_reply` is the code that followed the block.

`process_job` keeps the checks that apply to both phases and then picks one.
It is now 59 lines.

The statements moved without edits, apart from indentation and one `return`
that became the last line of its function. A script checked that both new
bodies parse to the same syntax trees as the code they came from.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
has_seen used binary_search, which is only correct on a sorted list.
The cursor JSON is shared with another producer, and a cursor whose
seen_ids arrive unsorted made already-posted ids read as unseen, so the
next slot could post them again. has_seen now does a linear contains,
which is order-independent and cheap at SEEN_CAP (200) ids.

Failing test first: unsorted_cursor_from_another_producer_still_dedups
reported "id 30 was posted but reads as unseen" before the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R8.12, as originally written, rolls a promoted configuration back when
the recent-20 mean is "5점 이상" (five or more) below the pre-promotion
baseline, and the property test's own doc says ">=5". The code and the
property oracle both used a strict comparison, so a drop of exactly 5.0
kept the worse configuration. watch_after_promotion now rolls back when
after_mean <= baseline - ROLLBACK_DROP, and the oracle matches.

This follows the written spec by default. If the owner wants strict
"more than 5", reverse the comparison in watch_after_promotion and in
the oracle at tests/self_improve_properties.rs and drop the new test.

Failing test first: improver_rolls_back_on_a_drop_of_exactly_five
(baseline 80, recent 75) reported "a 5-point drop must roll back".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
canonical_link_key lowercased the whole URL, so two different links
whose paths differ only by case (youtu.be/aBc and youtu.be/AbC, or a
case-sensitive query value) shared one ledger key. The second link read
as already delivered to the room and was silently never forwarded.

The key now lowercases only the scheme and authority, which URLs treat
case-insensitively, keeps the path, query, and fragment as written, and
still ignores trailing slashes. The R6.7 doc said "case-insensitive";
it now says scheme/host-case-insensitive. Property 21's UpperCase repeat
upper-cased the whole URL, so it now varies only the scheme and host.

Failing test first: canonical_link_key_keeps_path_case failed with both
keys equal to "https://youtu.be/abc".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ladder recorded an acquirer's own AcquireError::PermissionDenied as
StepAttempt::PermissionMissing, the same variant the screen-recording
gate uses when it skips the capture rung. acquire_miss_reason then told
the user screen recording was missing whenever any rung was denied, even
with screen recording granted, which points them at the wrong setting.

A rung's own denial is now StepAttempt::PermissionDenied, and its miss
reason says the image could not be accessed. PermissionMissing again
means only the screen-recording gate (R9.10).

Failing test first:
original_save_denial_is_not_reported_as_missing_screen_recording failed
with "화면 기록 권한이 없어 일부 이미지를 가져오지 못했어요." while
screen recording was granted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A post with more than MAX_IMAGES_PER_POST (10) images sends only the
first ten (R6.4), but deliver_to_room compared delivered images against
the full source count. Twelve images could never be "complete", so the
(room, link) was never written to the ledger, and every re-forward sent
the same post to the room again, which breaks once per (room, link)
(R6.7).

ImageTally.source now counts the images actually sent (the post's images
up to the cap). The dropped count is still reported on
LinkOutcome::Delivered and still named in a partial failure's reasons,
and miss_reasons no longer counts dropped images as missing ones. The
journal records the same capped count as the tally.

Two tests pinned the old rule and now assert the R6.7 contract:
forward_journals_and_reports_every_room_outcome (twelve images, ten
delivered) failed before the change with PartialFailure { source: 12,
delivered: 10 } where Sent { source: 10, delivered: 10 } is expected,
plus forward_over_ten_images_counts_dropped and property 19's oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SqliteDeliveryLedger::is_delivered returned query_row().is_ok(), so any
SQLite error (a locked, corrupt, or missing table) read as "not
delivered" and the forwarder sent the link to the room again. After a
complete send, `let _ = ledger.mark()` dropped a failed write, and the
room still reported Sent even though nothing recorded it, which defeats
once per (room, link) (R6.7, R6.14).

DeliveryLedger::is_delivered now returns Result<bool, LedgerError>. A
read error holds the room (no send) as Failed { code:
"ledger_read_failed" }. A failed mark after a complete send reports
Failed { code: "ledger_write_failed" }, documented as "the room has it,
forwarding again sends it twice", instead of Sent.

Failing tests first, against the real SQLite store with a fault injected
from a second connection:
- unreadable_ledger_holds_the_send (table dropped): 1 send where 0 is
  required.
- unwritable_ledger_does_not_report_sent (BEFORE INSERT trigger that
  aborts): reported Sent { source: 0, delivered: 0 }.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay runs unattended and ticks again on its own, so a store error
that reads as "not delivered" turns into repeated sends:
- SqliteRelayStore::is_delivered returned query_row().is_ok(), so any
  SQLite error read as "not delivered" and the room was sent the
  message.
- `let _ = ledger.mark()` dropped a failed write and settled the
  message as Committed, and the tick went on to send the next one.
- `let _ = cursor_store.save()` dropped a failed cursor write.
- A failed bump_mismatch became MISMATCH_STREAK_LIMIT, dropping the
  message as a final miss.
- A cursor that failed to load became the default cursor, which would
  replay the stream from the beginning.

RelayLedger::is_delivered and mismatch_streak now return Result, and
process_message propagates every store error. The first store error
latches the relay: it keeps the cursor where it was, journals
relay_store_failed, sends nothing more on that tick or any later one,
and reports the error in the new RelayTick::store_error. A cursor load
error latches the relay at construction. The cursor only advances in
memory after the save succeeds.

Failing tests first, against the real SQLite store with a fault injected
from a second connection:
- unreadable_ledger_holds_without_sending (table dropped): 1 send where
  0 is required.
- unwritable_ledger_stops_the_relay_after_one_send (BEFORE INSERT
  trigger that aborts, two queued messages): 2 sends where 1 is the
  limit, because the unrecorded first message counted as committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of send_reply

`send_reply` is the last check before a reply is typed into KakaoTalk.
Between its first and second readiness fence it ran three setup steps
inline: the composer allowlist write, the 30-key environment for the
`local-send` child, and the retried read-only preflight. The preflight
reported success through a `preflight` variable that the code tested after
the loop. A reader had to scroll through about 200 lines of setup to follow
the checks.

Move each step into its own function:

- `_allow_composer_draft` writes the reply into `composer-allowlist.json`.
- `_local_send_environment` returns the environment dict.
- `_send_preflight_ready` runs the preflight loop and returns whether the
  send may start. Each `break` that set `preflight` is now `return True`.
  The failure handling after the loop always returns False.

No other statement changed. An AST comparison shows that the composer block
and the environment dict are identical, and that the preflight loop differs
only in the removed flag. The environment dict is now built before the
preflight command list instead of after it. That list is a literal with no
side effects. A symtable check shows that no moved statement reads a
`send_reply` local that it is not passed.

Pin, with HOME set to an empty temporary directory, under a sandbox that
denies network, real-HOME writes, and Apple Events: the CI Python list plus
every module that imports the worker gives the same 953 results as base
533dbf6 (839 pass, 105 menubar bootstrap errors, 9 skips). The 218
test_auto_reply_cli_runtime tests that reach the worker and pass on a fresh
base all pass. The auto-reply-send-once queue tables are identical to the
ones at 533dbf6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R7.8, as originally written ("Retry only the missing images", with
MISS_RETRY_WINDOW_SECS = 30 as the retry window), was lost. 704923f
deleted the unused window constant, and 6ec71e1 rewrote the doc to match
the code, which re-acquired every image and re-sent the whole message,
text included, to each room on every retry. A message with more than
MAX_IMAGES (10) images could never count as complete, so every room got
its text three times before the message settled as a final miss.

Each (room, message) row in relay_ledger now records what the room has
received (RoomProgress: text_sent, an images_sent bitmask over the
capped images, the streak, and first_miss_at). A retry sends only the
missing images with an empty text, and acquires an image only if some
room still misses it. The images over the cap are dropped and reported,
and no longer block completion. A retry is allowed while the streak is
under MISMATCH_STREAK_LIMIT and less than MISS_RETRY_WINDOW_MS (30 s) has
passed since the first mismatch. RelaySender now documents that the text
may be empty on a retry and that images_delivered counts from the first
image. RelayLedger's bump_mismatch/mismatch_streak become record_miss and
progress; is_delivered is a provided method over progress.

This also fixes two smaller defects in the same code:
- The mismatch upsert hard-coded the limit 3 and always wrote timestamp
  0. The schema CHECK and the upsert now use MISMATCH_STREAK_LIMIT, and
  record_miss writes the real time, keeping the first miss's time for the
  window.
- An image-only message whose images all failed to acquire was sent with
  an empty body and no images. Such an attempt now sends nothing and
  counts as a mismatch.

Failing tests first (each ran against the previous code):
- image_retry_sends_only_the_missing_images: the second tick never
  committed.
- image_mismatch_retries_then_settles_final_after_three: sends were
  [("with images",2) x3] where [("with images",2), ("",2), ("",2)] is
  required.
- more_than_max_images_commits_once_the_capped_images_arrive: never
  committed.
- missing_images_are_retried_only_within_the_window: still MissRetrying
  30 s after the first miss.
- image_only_message_with_no_acquired_image_sends_nothing: 1 send where
  0 is required.
Property 19 now uses a sender that delivers on its first send only, and
asserts the text goes once and each retry carries exactly the missing
images. dropped_images_over_cap_report_and_never_commit pinned the old
rule and is replaced.

The relay_ledger columns are new; no production database exists yet
(the relay has no caller), so there is no migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RelaySendOutcome::Failed shared the Fenced arm, so one failed send
settled the message as MissFinal, the cursor moved past it, and the room
never received it. A fence is a deliberate, final refusal (R7.15); a
failed send is usually transient.

A failed send now leaves the room unsettled and the message
MissRetrying, so the cursor stays put and the room is tried again on the
next tick. It does not count as an image mismatch, so it does not use up
the three-strike streak or start the 30 s window. Fenced rooms are
unchanged.

Trade-off for the owner: a room that fails on every send now holds its
pair's queue (messages are relayed one at a time, R7.9) until the send
succeeds, rather than dropping messages. Each held tick journals
relay_miss_retry.

Failing test first: failed_send_holds_the_message_for_the_next_tick
reported the cursor at 1 (MissFinal) after the first, failed tick, where
it must stay at 0 and commit on the second tick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both modules were added with the Python runtime fixes and pass on a
fresh checkout under an empty HOME.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a timeout

try_aside mapped every health-check error other than NotInstalled to
SwitchReason::HealthTimeout, so an immediate "connection refused" told
the user "Aside 연결 확인이 늦어" (the check was slow). That points the
owner at a slow Aside when Aside is actually down or misconfigured.

A new SwitchReason::HealthFailed ("Aside 연결 확인에 실패해 ...") covers a
check that answered with an error. HealthTimeout stays for a check that
did not answer within three seconds (R11.2), and an Unauthorized health
answer maps to SecretUnavailable like the same error mid-collection.

Failing test first: a_failed_health_check_is_not_reported_as_a_timeout
reported "Aside 연결 확인이 늦어 다른 방법으로 가져왔어요" for a
Transport("connection refused") health error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FallbackCollector turned the whole HTTP response body into the
Collected body, so a huge or hostile page was copied in full and then
masked and summarized in full downstream, although a forward summary
keeps at most 700 characters.

The fallback now keeps at most MAX_FALLBACK_BODY_BYTES (1 MiB), cut
before any UTF-8 sequence the bound would split, so no character is
mangled into U+FFFD. The 1 MiB figure is a judgment call, not a spec
value. It is far above anything the summary uses.

This bounds only the copy into the domain type. The response is already
in memory when NetworkPort returns it, so the real read bound belongs in
the NetworkPort adapter (src/ports, outside this change; no production
adapter exists yet).

Failing test first: fallback_bounds_the_page_body_on_a_character_boundary
kept all 6291456 bytes of a 6291456-byte page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing it

RoutingCollector::collect computed the R11.8 DeletionReport and threw it
away (`let _report = scope.finish()`). R11.8 as originally written sends
that report to the History window. If the raw page data could not be
deleted, or the deletion took more than five seconds, nothing said so,
and the collection still reported success.

A deletion that fails or misses the bound now returns
CollectFailure::TempDataNotDeleted(report), replacing any other outcome.
The forwarder already turns a collection failure into zero sends (R6.8)
and a collect_failed journal record, which is what the History window
reads.

Choice for the owner: a good collection is discarded when its raw data
lingers. That keeps the privacy breach from going unnoticed, at the
cost of not forwarding that link. A successful, timely deletion is still
not journaled.

Failing test first: undeleted_temp_data_is_reported_not_dropped (a page
collector leaves a file in a read-only directory under the scope, so the
real deletion fails) got Ok(Collected { .. }).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AsideCollector checked the three-second health bound (R11.2) and the
ten-second collection bound (R11.11) only after the transport call
returned, measured on the injected clock. A transport that never returns
therefore blocked the collector, the router's twenty-second bound
(R11.12), and the forward behind it, forever.

The collector cannot interrupt a blocking call: it borrows the transport
and lends it the TempScope, and Rust cannot cancel a thread, so a
watchdog would need 'static ownership of both and would still leak the
stuck thread. The bound now travels with each call instead.
AsideTransport::health and fetch take `bound: Duration` (3 s and 10 s),
and the trait documents that the transport must give up by then (for
example as its HTTP client timeout). The post-hoc clock checks stay and
still classify a late answer as a timeout.

Not fixed here: FallbackCollector calls NetworkPort::request, whose
HttpRequest has no timeout field. That lives in src/ports, outside this
change, so a hung fallback request still blocks. No production
AsideTransport or NetworkPort exists yet.

Failing test first: aside_hands_each_transport_call_its_time_bound could
not compile against the old trait (E0050, health has 2 parameters and
fetch 5, so no bound could reach the transport). It passes now with
[3 s, 10 s].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@twoimo
twoimo changed the base branch from stability/2-send-path-followups to main November 16, 2026 00:31
@twoimo twoimo closed this Nov 16, 2026
@twoimo twoimo reopened this Nov 16, 2026
@twoimo twoimo closed this Nov 16, 2026
@twoimo twoimo reopened this Nov 16, 2026
@twoimo
twoimo merged commit 6de6924 into main Nov 16, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant