Repository navigation
fix(auto-reply): record ax_mutation_authorized on the job so a reply posts once - #52
Merged
Merged
Conversation
_sanitize_outbound_reaction resamples every ㅋ run with the global RNG, so an allowed "ㅋㅋㅋㅋ 그건 좀" comes back as a 3, 4 or 5 run and the same draft gives different replies. That is why test_laughter_allowance_follows_clean_profile_frequency fails 18 of 30 runs on stability-wave1. Both repro tests are marked expectedFailure until the fix lands. A third test keeps the resampler's purpose: exact ㅋㅋㅋ drafts must still vary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a journal rollover A reply that reached ax_mutation_authorized and ended delivery_unknown keeps its only send evidence in pipeline_transitions. Once 4096 newer rows from other room messages arrive, the cap trigger deletes that evidence, the next recovery pass reschedules the job, and the worker posts the same reply again. This drives the real send path to delivery_unknown, journals 4096 other room messages the way db-watch does, then runs the worker loop's recovery, give-up and claim steps. It fails today with two local-send calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_resample_laughter_runs exists to break the model's habit of always writing exactly three ㅋ. It drew every run of three or more from the global RNG, so an allowed four-run reply came back shortened or lengthened and one draft gave different replies on each parse. Resample only exact three-runs, and seed the draw with the draft text. A run the model already varied stays as written, and the same draft always sanitizes to the same reply. Different drafts still vary. test_laughter_allowance_follows_clean_profile_frequency: 12/30 passed before, 30/30 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker installs no SIGTERM handler, so SIGTERM kills it without running any finally block. Model runners start in their own session (isolate_group=True), so neither the Rust group sweep nor the Python supervisor reaches them, and they outlive the worker. The harness drives worker_main with every collaborator faked except one real isolated runner that ignores SIGTERM. It sends SIGTERM twice, as the group sweep and then the supervisor's stop() do. The second test pins that _open_pinned_link does not swallow a shutdown raised during one address and carry on to the next. Both are expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIGTERM killed the worker outright, so _run_bounded_process never ran its finally block. Model runners started with isolate_group=True live in their own session, so they outlived the worker. Neither the Rust group sweep nor the Python supervisor's terminate() can reach them. worker_main now turns SIGTERM into SystemExit(143), so the bounded runner's cleanup (TERM, then KILL of the isolated group) and worker_main's own finally both run. On shutdown the worker gets SIGTERM twice: once from the Rust group sweep and once from the supervisor's stop(). A second SystemExit raised during cleanup skipped the KILL escalation 3 of 3 times, so later signals are ignored once shutdown has started. SIGKILL after the supervisors' 3 s grace stays the backstop. The send CLI still runs in the worker's group, and the sweep still owns it. _open_pinned_link caught BaseException per address, so a shutdown raised while connecting to one address was swallowed when the next succeeded. It now re-raises anything that is not an Exception. SIGTERM test: 30/30 after the fix. Before it, the runner survived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_next_geeknews_digest defaults persist_seen=True, so a plain call writes the chosen ids as seen before anything is sent. AGENTS.md allows seen and slots to be persisted only after a confirmed send. Every live caller passes False, so the default is a trap for the next caller. expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds tests/auto_reply_hermetic.py. Per test it gives an empty temporary HOME, clears every OPENKAKAO_* variable, and restores the environment afterwards. Per module it installs guards that refuse any outbound connection that is not loopback to a port this process bound, refuse the live model port 127.0.0.1:11234 outright, and refuse any file open or sqlite connect under the real HOME outside the repository checkout. A refused call raises and is recorded; each test fails in cleanup when a violation was recorded, even when production code swallowed the exception. Co-Authored-By: GPT-6.1 Sol via opencodex <noreply@openai.com>
…drift - Wire the harness into setUpModule and AutoReplyCliRuntimeTests.setUp: each test gets its own HOME, a clean OPENKAKAO_* environment, and guards. - Pin the failure-record endpoint probe to "unreachable". It opened a real loopback socket, so what a failure row records depended on the machine. - Seed an empty, owner-only context decision store in the temporary HOME. The runtime refuses to read a missing store, and an operator machine always has one. - Point QUEUE and LOCK at the same room directory (_point_queue), as production derives both from one state root. Relocating only the queue tripped the lock parent check. - Treat the room as current at each inbound event's own tail (_room_is_current) instead of requiring the supervisor's db-watch fence, and record the analysis watermark on hand-built scheduled jobs, as production does at schedule time. - Stub the durable skip audit where its store read-back is not hermetic, pin the laughter and clarification random draws, pin the GeekNews slot clock for proactive jobs, and pin BIN where no installed app exists. - Fix the enrolled-room db-watch fixture: its queue lives in the room state root and is created by the supervisor fixture before the watcher attaches. - Leave 15 tests failing with their reasons. No assertion was weakened and nothing is skipped. Co-Authored-By: GPT-6.1 Sol via opencodex <noreply@openai.com>
_next_geeknews_digest wrote the chosen ids as seen whenever a caller forgot persist_seen=False, before any send. AGENTS.md persists seen and slots only after a confirmed send. Both live callers passed False, so the write branch is deleted with the parameter. Only _mark_geeknews_digest_confirmed and the operator's confirmed path record posts now. test_geeknews_digest_posts_unseen_feed_items_once relied on the trap to show an item is posted once. It now confirms the first digest before selecting again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rsor The room worker and the host slot CLI (launchd com.openkakao.nimda-geeknews runs auto-reply-worker.py --geeknews --send at 08:40, 12:35 and 19:50) both read-modify-write rooms/<id>/geeknews-rss-cursor.json with no lock. When two confirmed posts interleave, the later write drops the earlier ids, and the next slot posts them again. The repro slows the first writer at os.replace so the interleaving is deterministic. expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes record GeekNews posts in rooms/<id>/geeknews-rss-cursor.json. One is the room worker after a confirmed send. The other is the host slot CLI (auto-reply-worker.py --geeknews --send from launchd). Each loaded the cursor, merged in its ids and replaced the file, so overlapping writes dropped one post's ids. _store_geeknews_seen_ids now does the load, merge and replace inside _private_lock on .geeknews-cursor.lock next to the cursor. That is the owner-only flock helper the worker already uses for its state lock. The store owns the merge, so callers pass only the post they are recording. The operator's --mark-slot used a second, separate read-modify-write. It now goes through the same locked store as posted_slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_mark_geeknews_digest_confirmed is called with now=time.time() and derives the posted-slot marker from that stamp. A morning post confirmed inside the lunch window marks lunch as posted, so lunch is skipped and the morning slot stays open. expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A GeekNews job carries sent_at, the moment it was enqueued inside an open slot. _geeknews_slot_has_attempt already uses that stamp to decide which slot a job occupies. The confirmed-post marker now comes from the same stamp, so a late confirmation can no longer close the next slot and leave its own open. The confirmation time still stamps updated_at, which the menubar reads as the confirmation age. A forced post sent outside every slot marks none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only record that a job reached ax_mutation_authorized lived in pipeline_transitions, a rolling window that the cap trigger trims to the newest 4096 rows. The job row is the work authority, so the fact now lives there: reply_jobs.ax_mutation_authorized (0 or 1). The insert and status triggers set it in the same statement that puts the job in sending, the way the status trigger already keeps attempt_no, and no later status clears it. A row in sending therefore always carries it, however it got there. Queue schema v3 adds the column. The supervisor's open_queue(create=True) migrates an exact v2 queue in one IMMEDIATE transaction: add the column, backfill it, replace both reply_jobs triggers, set user_version 3. A fault rolls back to the exact v2 queue, and a second run finds v3 and only validates it. A fresh or v0 queue takes the same v3 step after the existing v2 one. The backfill marks every job whose status still leaves a send possible (sending, delivery_unknown, reconcile_required, poison) as well as every job whose journal still shows an attempt. A v2 delivery_unknown job may already have lost its journal rows to the cap, so it counts as possibly sent. The Rust stopped-clean validator accepts v0, v2 and v3, so a queue written before the upgrade still resumes, and checks the v3 triggers and column values. The worker still reads the journal; the next change moves it to the column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… longer shows leftover_queue_has_unknown_send decides whether an unclean stop may resume. It counts a delivery_unknown job as an unknown send only while the job's AX rows are still in pipeline_transitions. On a v3 queue the job row says whether Return may have been pressed, and the cap can no longer erase that. This fails today: the authorized job without journal rows is let through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
geeknews_operator_cli has three defects: - It runs local-send with no timeout, so a hung AX send blocks the slot forever. - It posts to --chat by window title but verifies the room id parsed from --queue. With a mismatch it posts to one room, finds nothing in the other, exits 3 and leaves the cursor alone, so the next slot posts again. - --queue defaults to a hard-coded room (417780809780519), which pairs any --chat with that room. The fakes stand in only for the feed fetch, local-read and local-send. A post lands in the room whose title it was sent to. The confirmed-send test passes today and keeps the happy path covered. The others are expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pressed leftover_pre_send_unknown_skip_fields kept a delivery_unknown job from being sent again only while leftover_unknown_has_ax_mutation still found the job's AX rows in pipeline_transitions. Once the journal cap deleted them, the next recovery pass made the job scheduled again and _queue_reconciliation_blockers stopped counting it, so the worker posted the same reply a second time. Every reader of that fact now asks the job row: - leftover_unknown_has_ax_mutation reads reply_jobs.ax_mutation_authorized; - _queue_reconciliation_blockers exempts a formed reply only when the row never reached sending; - the Rust leftover_queue_has_unknown_send does the same on a v3 queue and keeps the journal rule for a v2 queue the supervisor has not upgraded yet. A pre_send ready row alone no longer counts as a send attempt. local-send is spawned only after the sending commit, so such a job cannot have pressed Return. The journal rollover repro now posts once and stays fenced, and the leftover repro occupies the room. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host slot CLI could hang forever on local-send. It could also post to one room and verify another, then post again at the next slot. - local-send now runs with a 60 s timeout, the worker's own send bound. On timeout the CLI still reads the room back and records the post only if it landed. Otherwise it exits 3 and leaves the cursor alone. - The window title and the readback room come from one room-catalog entry: the room id in the --queue path and that entry's title. _filter_catalog_selectors attests that pair as bind:<id>:<title>. The local DB cannot name an unnamed group room such as 325472527151234. --chat must be that title or that id. Anything else exits 4 unsent. An id now sends to the bound title, not to a window called "id:<n>". - --queue is required. Its default was a hard-coded room (417780809780519) that silently paired with any --chat. The launchd slot script passes --chat, --bin and --queue explicitly. Checked read-only against the live catalog, its arguments resolve to room 325472527151234. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The model runner's PATH starts with /Users/twoimo/.bun/bin, so the gjc runner only finds bun under that one account. expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ectory The runner env already sets HOME from Path.home(), but PATH named /Users/twoimo/.bun/bin. It now uses Path.home() / ".bun" / "bin". For the real account it is byte-identical to the old literal (checked), and under any other account the gjc runner now finds bun. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_store_geeknews_seen_ids swallows OSError from the cursor write. After a confirmed send the post's ids are never recorded, nothing says so, and the next slot posts the same TOP5 again. The worker and the host slot CLI both hit this. The repros make the cursor path a directory, a real write fault with no mock. They pin four things: - the worker logs an alert naming the ids - the worker's next selection skips ids its queue shows as sent - the operator exits non-zero - the operator skips ids that its room readback shows as our own posts expectedFailure until the fix lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lder or duplicate row When local-send answers accepted_unconfirmed, send_reply looks for the reply itself and accepts the newest exact self row after the inbound message. A row the account wrote with the same text before the send passes that test, and so does one of two delivered copies. local-send now confirms only from rows newer than its pre-send tail and rejects two matches. FakeCli reports that tail the way pre_send_tail_log_id computes it. Both new pins fail today: the job is recorded as sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_store_geeknews_seen_ids swallowed every OSError from the cursor write. After a confirmed send the post was never recorded and nothing reported it, so the next slot posted the same TOP5 again. The store now raises, and each caller handles the failure for its own path. Surface: - The worker records the post inside send_reply, after the local DB has confirmed it. Raising there would exit the worker with the job left in `sending`, and nothing would stop the re-post. It logs a [geeknews] alert with the error and the ids instead. - The host slot CLI exits 5 (confirmed, not recorded) with the ids. launchd logs the exit status. Don't re-pick: digest selection also skips ids from a second durable record of past posts. - The worker reads the proactive_ids of this room's `sent` GeekNews jobs. The queue keeps terminal rows for 30 days, longer than an item stays in the feed. - The CLI reads the room before choosing items, and skips topics that our own messages in the 100-row baseline already linked. It already took that baseline for confirmation. A link someone else shared does not count. Fencing until the cursor is writable was rejected. The post that failed to record would still be forgotten once the cursor became writable again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n its pre-send tail When local-send answered accepted_unconfirmed, send_reply looked for the reply in the newest 8 rows and took the newest exact self row after the inbound message. A row the account had written with the same text before the send counted as this send's proof, and one of two delivered copies counted too, so a duplicate was recorded as sent with a local_db_confirmed checkpoint. local-send already confirms only from rows newer than the room tail it read before touching the composer, and treats two matches as a duplicate. A worker-bound accepted_unconfirmed result now carries that tail as pre_send_tail_log_id, and the fallback applies the same rule from it: exactly one own text row newer than the tail, on a page that reaches back to the tail. Without the tail it confirms nothing and the job stays delivery_unknown. The checks that ask whether a reply is already in the room keep their inbound floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…composer allowlist composer-allowlist.json lists the texts the worker may have typed, and local-send replaces a composer holding one of them. send_reply writes the reply there before the read-only preflight and before the last hold check, so a reply that is never typed still authorizes overwriting a composer that holds the same text. This fails today: after a failed preflight and a held send, the allowlist already lists both replies. A reply that does reach local-send must still be listed when local-send starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests.test_auto_reply_worker_stability holds the regression tests for the GeekNews cursor and operator CLI, worker SIGTERM cleanup and deterministic laughter sanitizing. It is hermetic: temp state roots, fake feed and local-read/local-send, and one real fake model runner process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s about to be typed send_reply appended the reply to composer-allowlist.json before the read-only preflight and before _pre_mutation_send_hold. local-send replaces a composer holding an allowlisted text, so a reply that was held, or whose preflight failed, still let local-send overwrite a composer holding the same words, for example a draft the account typed by hand. The append now runs after the sending commit and immediately before the local-send child that may type the reply, so the list names only texts the worker may have put in the composer. The preflight never reads the list: it reports any non-empty composer as composer_occupied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
twoimo
changed the base branch from
stability/3-worker-refactor-and-dormant-modules
to
main
November 16, 2026 00:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
An automatic reply could be posted twice. The only record that Return was pressed lived in a 4,096-row rolling journal; once newer rows evicted it, recovery rescheduled the job and sent it again. A TLA+ model check over tables taken from the real queue code found this and a replay reproduced it.
What changed
reply_jobs.ax_mutation_authorizedis set by the queue triggers in the same statement that enterssending; recovery reads it instead of the journal. Schema v3 with a one-transaction v2 migration run only by the supervisor; jobs that may have sent are never rescheduled.Scope
Tradeoffs
Blast Radius
An older runtime refuses v3 queues, so rolling back after the upgrade stops auto-reply (fails closed). After the upgrade, any
delivery_unknownrow with a draft blocks its room until it is reconciled, even if it never reachedsending; the three live queues hold no such row today. A pre-upgradedelivery_unknownjob without a draft keeps its old recovery.Verification
cargo test1,168 passed; CI Python list 1,015 passed under the sandbox guard.🤖 Generated with Claude Code