Skip to content

fix(auto-reply): record ax_mutation_authorized on the job so a reply posts once - #52

Merged
twoimo merged 31 commits into
mainfrom
stability/4-send-once-and-geeknews
Nov 16, 2026
Merged

twoimo merged 31 commits into
mainfrom
stability/4-send-once-and-geeknews

Conversation

@twoimo

@twoimo twoimo commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Why

An automatic reply could be posted twice. The only record that Return was pressed lived in a 4,096-row rolling journal; once newer rows evicted it, recovery rescheduled the job and sent it again. A TLA+ model check over tables taken from the real queue code found this and a replay reproduced it.

What changed

  • reply_jobs.ax_mutation_authorized is set by the queue triggers in the same statement that enters sending; recovery reads it instead of the journal. Schema v3 with a one-transaction v2 migration run only by the supervisor; jobs that may have sent are never rescheduled.
  • The Python fallback confirmation accepts only rows newer than the pre-send tail; the composer allowlist is written only right before typing.
  • GeekNews: no silent loss of a confirmed post, slot marker from the event, locked cursor writes, operator send with a timeout and one verified room, SIGTERM cleanup in the worker, deterministic laughter resampling, hermetic runtime test harness.

Scope

  • The migration runs on real queues only after a new runtime is baked and installed.

Tradeoffs

  • Exempting live jobs from the journal cap was rejected: it breaks the cap both validators enforce and keeps a safety fact in a log.

Blast Radius

An older runtime refuses v3 queues, so rolling back after the upgrade stops auto-reply (fails closed). After the upgrade, any delivery_unknown row with a draft blocks its room until it is reconciled, even if it never reached sending; the three live queues hold no such row today. A pre-upgrade delivery_unknown job without a draft keeps its old recovery.

Verification

  • Model check: journal-cap config passes (168,104 states); the real-code probe keeps the job after 64 recovery passes with all its journal rows evicted. A scratch revert of the fix is caught.
  • cargo test 1,168 passed; CI Python list 1,015 passed under the sandbox guard.

🤖 Generated with Claude Code

twoimo and others added 30 commits November 16, 2026 03:49
_sanitize_outbound_reaction resamples every ㅋ run with the global RNG,
so an allowed "ㅋㅋㅋㅋ 그건 좀" comes back as a 3, 4 or 5 run and the
same draft gives different replies. That is why
test_laughter_allowance_follows_clean_profile_frequency fails 18 of 30
runs on stability-wave1.

Both repro tests are marked expectedFailure until the fix lands. A third
test keeps the resampler's purpose: exact ㅋㅋㅋ drafts must still vary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a journal rollover

A reply that reached ax_mutation_authorized and ended delivery_unknown keeps
its only send evidence in pipeline_transitions. Once 4096 newer rows from
other room messages arrive, the cap trigger deletes that evidence, the next
recovery pass reschedules the job, and the worker posts the same reply again.

This drives the real send path to delivery_unknown, journals 4096 other room
messages the way db-watch does, then runs the worker loop's recovery, give-up
and claim steps. It fails today with two local-send calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_resample_laughter_runs exists to break the model's habit of always
writing exactly three ㅋ. It drew every run of three or more from the
global RNG, so an allowed four-run reply came back shortened or
lengthened and one draft gave different replies on each parse.

Resample only exact three-runs, and seed the draw with the draft text.
A run the model already varied stays as written, and the same draft
always sanitizes to the same reply. Different drafts still vary.

test_laughter_allowance_follows_clean_profile_frequency: 12/30 passed
before, 30/30 after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker installs no SIGTERM handler, so SIGTERM kills it without
running any finally block. Model runners start in their own session
(isolate_group=True), so neither the Rust group sweep nor the Python
supervisor reaches them, and they outlive the worker. The harness drives
worker_main with every collaborator faked except one real isolated
runner that ignores SIGTERM. It sends SIGTERM twice, as the group sweep
and then the supervisor's stop() do.

The second test pins that _open_pinned_link does not swallow a shutdown
raised during one address and carry on to the next.

Both are expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SIGTERM killed the worker outright, so _run_bounded_process never ran
its finally block. Model runners started with isolate_group=True live in
their own session, so they outlived the worker. Neither the Rust group
sweep nor the Python supervisor's terminate() can reach them.

worker_main now turns SIGTERM into SystemExit(143), so the bounded
runner's cleanup (TERM, then KILL of the isolated group) and
worker_main's own finally both run. On shutdown the worker gets SIGTERM
twice: once from the Rust group sweep and once from the supervisor's
stop(). A second SystemExit raised during cleanup skipped the KILL
escalation 3 of 3 times, so later signals are ignored once shutdown has
started. SIGKILL after the supervisors' 3 s grace stays the backstop.
The send CLI still runs in the worker's group, and the sweep still owns it.

_open_pinned_link caught BaseException per address, so a shutdown raised
while connecting to one address was swallowed when the next succeeded.
It now re-raises anything that is not an Exception.

SIGTERM test: 30/30 after the fix. Before it, the runner survived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_next_geeknews_digest defaults persist_seen=True, so a plain call writes
the chosen ids as seen before anything is sent. AGENTS.md allows seen and
slots to be persisted only after a confirmed send. Every live caller
passes False, so the default is a trap for the next caller.

expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds tests/auto_reply_hermetic.py. Per test it gives an empty temporary HOME,
clears every OPENKAKAO_* variable, and restores the environment afterwards.
Per module it installs guards that refuse any outbound connection that is not
loopback to a port this process bound, refuse the live model port
127.0.0.1:11234 outright, and refuse any file open or sqlite connect under the
real HOME outside the repository checkout. A refused call raises and is
recorded; each test fails in cleanup when a violation was recorded, even when
production code swallowed the exception.

Co-Authored-By: GPT-6.1 Sol via opencodex <noreply@openai.com>
…drift

- Wire the harness into setUpModule and AutoReplyCliRuntimeTests.setUp: each
  test gets its own HOME, a clean OPENKAKAO_* environment, and guards.
- Pin the failure-record endpoint probe to "unreachable". It opened a real
  loopback socket, so what a failure row records depended on the machine.
- Seed an empty, owner-only context decision store in the temporary HOME.
  The runtime refuses to read a missing store, and an operator machine always
  has one.
- Point QUEUE and LOCK at the same room directory (_point_queue), as production
  derives both from one state root. Relocating only the queue tripped the
  lock parent check.
- Treat the room as current at each inbound event's own tail (_room_is_current)
  instead of requiring the supervisor's db-watch fence, and record the analysis
  watermark on hand-built scheduled jobs, as production does at schedule time.
- Stub the durable skip audit where its store read-back is not hermetic, pin
  the laughter and clarification random draws, pin the GeekNews slot clock for
  proactive jobs, and pin BIN where no installed app exists.
- Fix the enrolled-room db-watch fixture: its queue lives in the room state
  root and is created by the supervisor fixture before the watcher attaches.
- Leave 15 tests failing with their reasons. No assertion was weakened and
  nothing is skipped.

Co-Authored-By: GPT-6.1 Sol via opencodex <noreply@openai.com>
_next_geeknews_digest wrote the chosen ids as seen whenever a caller
forgot persist_seen=False, before any send. AGENTS.md persists seen and
slots only after a confirmed send. Both live callers passed False, so
the write branch is deleted with the parameter. Only
_mark_geeknews_digest_confirmed and the operator's confirmed path record
posts now.

test_geeknews_digest_posts_unseen_feed_items_once relied on the trap to
show an item is posted once. It now confirms the first digest before
selecting again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rsor

The room worker and the host slot CLI (launchd com.openkakao.nimda-geeknews
runs auto-reply-worker.py --geeknews --send at 08:40, 12:35 and 19:50)
both read-modify-write rooms/<id>/geeknews-rss-cursor.json with no
lock. When two confirmed posts interleave, the later write drops the
earlier ids, and the next slot posts them again. The repro slows the
first writer at os.replace so the interleaving is deterministic.

expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two processes record GeekNews posts in rooms/<id>/geeknews-rss-cursor.json.
One is the room worker after a confirmed send. The other is the host slot
CLI (auto-reply-worker.py --geeknews --send from launchd). Each loaded
the cursor, merged in its ids and replaced the file, so overlapping
writes dropped one post's ids.

_store_geeknews_seen_ids now does the load, merge and replace inside
_private_lock on .geeknews-cursor.lock next to the cursor. That is the
owner-only flock helper the worker already uses for its state lock. The
store owns the merge, so callers pass only the post they are recording.
The operator's --mark-slot used a second, separate read-modify-write. It
now goes through the same locked store as posted_slot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_mark_geeknews_digest_confirmed is called with now=time.time() and
derives the posted-slot marker from that stamp. A morning post
confirmed inside the lunch window marks lunch as posted, so lunch is
skipped and the morning slot stays open.

expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A GeekNews job carries sent_at, the moment it was enqueued inside an
open slot. _geeknews_slot_has_attempt already uses that stamp to decide
which slot a job occupies. The confirmed-post marker now comes from the
same stamp, so a late confirmation can no longer close the next slot
and leave its own open. The confirmation time still stamps updated_at,
which the menubar reads as the confirmation age. A forced post sent
outside every slot marks none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only record that a job reached ax_mutation_authorized lived in
pipeline_transitions, a rolling window that the cap trigger trims to the
newest 4096 rows. The job row is the work authority, so the fact now lives
there: reply_jobs.ax_mutation_authorized (0 or 1). The insert and status
triggers set it in the same statement that puts the job in sending, the way
the status trigger already keeps attempt_no, and no later status clears it.
A row in sending therefore always carries it, however it got there.

Queue schema v3 adds the column. The supervisor's open_queue(create=True)
migrates an exact v2 queue in one IMMEDIATE transaction: add the column,
backfill it, replace both reply_jobs triggers, set user_version 3. A fault
rolls back to the exact v2 queue, and a second run finds v3 and only
validates it. A fresh or v0 queue takes the same v3 step after the existing
v2 one.

The backfill marks every job whose status still leaves a send possible
(sending, delivery_unknown, reconcile_required, poison) as well as every job
whose journal still shows an attempt. A v2 delivery_unknown job may already
have lost its journal rows to the cap, so it counts as possibly sent.

The Rust stopped-clean validator accepts v0, v2 and v3, so a queue written
before the upgrade still resumes, and checks the v3 triggers and column
values. The worker still reads the journal; the next change moves it to the
column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… longer shows

leftover_queue_has_unknown_send decides whether an unclean stop may resume.
It counts a delivery_unknown job as an unknown send only while the job's AX
rows are still in pipeline_transitions. On a v3 queue the job row says
whether Return may have been pressed, and the cap can no longer erase that.
This fails today: the authorized job without journal rows is let through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
geeknews_operator_cli has three defects:

- It runs local-send with no timeout, so a hung AX send blocks the slot
  forever.
- It posts to --chat by window title but verifies the room id parsed
  from --queue. With a mismatch it posts to one room, finds nothing in
  the other, exits 3 and leaves the cursor alone, so the next slot posts
  again.
- --queue defaults to a hard-coded room (417780809780519), which pairs
  any --chat with that room.

The fakes stand in only for the feed fetch, local-read and local-send. A
post lands in the room whose title it was sent to. The confirmed-send
test passes today and keeps the happy path covered. The others are
expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pressed

leftover_pre_send_unknown_skip_fields kept a delivery_unknown job from being
sent again only while leftover_unknown_has_ax_mutation still found the job's
AX rows in pipeline_transitions. Once the journal cap deleted them, the next
recovery pass made the job scheduled again and _queue_reconciliation_blockers
stopped counting it, so the worker posted the same reply a second time.

Every reader of that fact now asks the job row:
- leftover_unknown_has_ax_mutation reads reply_jobs.ax_mutation_authorized;
- _queue_reconciliation_blockers exempts a formed reply only when the row
  never reached sending;
- the Rust leftover_queue_has_unknown_send does the same on a v3 queue and
  keeps the journal rule for a v2 queue the supervisor has not upgraded yet.

A pre_send ready row alone no longer counts as a send attempt. local-send is
spawned only after the sending commit, so such a job cannot have pressed
Return.

The journal rollover repro now posts once and stays fenced, and the leftover
repro occupies the room.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host slot CLI could hang forever on local-send. It could also post
to one room and verify another, then post again at the next slot.

- local-send now runs with a 60 s timeout, the worker's own send bound.
  On timeout the CLI still reads the room back and records the post only
  if it landed. Otherwise it exits 3 and leaves the cursor alone.
- The window title and the readback room come from one room-catalog
  entry: the room id in the --queue path and that entry's title.
  _filter_catalog_selectors attests that pair as bind:<id>:<title>. The
  local DB cannot name an unnamed group room such as 325472527151234.
  --chat must be that title or that id. Anything else exits 4 unsent. An
  id now sends to the bound title, not to a window called "id:<n>".
- --queue is required. Its default was a hard-coded room
  (417780809780519) that silently paired with any --chat. The launchd
  slot script passes --chat, --bin and --queue explicitly. Checked
  read-only against the live catalog, its arguments resolve to room
  325472527151234.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The model runner's PATH starts with /Users/twoimo/.bun/bin, so the gjc
runner only finds bun under that one account. expectedFailure until the
fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ectory

The runner env already sets HOME from Path.home(), but PATH named
/Users/twoimo/.bun/bin. It now uses Path.home() / ".bun" / "bin". For
the real account it is byte-identical to the old literal (checked), and
under any other account the gjc runner now finds bun.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_store_geeknews_seen_ids swallows OSError from the cursor write. After a
confirmed send the post's ids are never recorded, nothing says so, and
the next slot posts the same TOP5 again. The worker and the host slot
CLI both hit this.

The repros make the cursor path a directory, a real write fault with no
mock. They pin four things:
- the worker logs an alert naming the ids
- the worker's next selection skips ids its queue shows as sent
- the operator exits non-zero
- the operator skips ids that its room readback shows as our own posts

expectedFailure until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lder or duplicate row

When local-send answers accepted_unconfirmed, send_reply looks for the reply
itself and accepts the newest exact self row after the inbound message. A row
the account wrote with the same text before the send passes that test, and
so does one of two delivered copies. local-send now confirms only from rows
newer than its pre-send tail and rejects two matches.

FakeCli reports that tail the way pre_send_tail_log_id computes it. Both new
pins fail today: the job is recorded as sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_store_geeknews_seen_ids swallowed every OSError from the cursor write.
After a confirmed send the post was never recorded and nothing reported
it, so the next slot posted the same TOP5 again. The store now raises,
and each caller handles the failure for its own path.

Surface:
- The worker records the post inside send_reply, after the local DB has
  confirmed it. Raising there would exit the worker with the job left in
  `sending`, and nothing would stop the re-post. It logs a
  [geeknews] alert with the error and the ids instead.
- The host slot CLI exits 5 (confirmed, not recorded) with the ids.
  launchd logs the exit status.

Don't re-pick: digest selection also skips ids from a second durable
record of past posts.
- The worker reads the proactive_ids of this room's `sent` GeekNews jobs.
  The queue keeps terminal rows for 30 days, longer than an item stays in
  the feed.
- The CLI reads the room before choosing items, and skips topics that our
  own messages in the 100-row baseline already linked. It already took
  that baseline for confirmation. A link someone else shared does not
  count.

Fencing until the cursor is writable was rejected. The post that failed
to record would still be forgotten once the cursor became writable again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n its pre-send tail

When local-send answered accepted_unconfirmed, send_reply looked for the
reply in the newest 8 rows and took the newest exact self row after the
inbound message. A row the account had written with the same text before the
send counted as this send's proof, and one of two delivered copies counted
too, so a duplicate was recorded as sent with a local_db_confirmed checkpoint.

local-send already confirms only from rows newer than the room tail it read
before touching the composer, and treats two matches as a duplicate. A
worker-bound accepted_unconfirmed result now carries that tail as
pre_send_tail_log_id, and the fallback applies the same rule from it:
exactly one own text row newer than the tail, on a page that reaches back to
the tail. Without the tail it confirms nothing and the job stays
delivery_unknown. The checks that ask whether a reply is already in the room
keep their inbound floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…composer allowlist

composer-allowlist.json lists the texts the worker may have typed, and
local-send replaces a composer holding one of them. send_reply writes the
reply there before the read-only preflight and before the last hold check, so
a reply that is never typed still authorizes overwriting a composer that
holds the same text.

This fails today: after a failed preflight and a held send, the allowlist
already lists both replies. A reply that does reach local-send must still be
listed when local-send starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests.test_auto_reply_worker_stability holds the regression tests for
the GeekNews cursor and operator CLI, worker SIGTERM cleanup and
deterministic laughter sanitizing. It is hermetic: temp state roots,
fake feed and local-read/local-send, and one real fake model runner
process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s about to be typed

send_reply appended the reply to composer-allowlist.json before the
read-only preflight and before _pre_mutation_send_hold. local-send replaces a
composer holding an allowlisted text, so a reply that was held, or whose
preflight failed, still let local-send overwrite a composer holding the same
words, for example a draft the account typed by hand.

The append now runs after the sending commit and immediately before the
local-send child that may type the reply, so the list names only texts the
worker may have put in the composer. The preflight never reads the list: it
reports any non-empty composer as composer_occupied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@twoimo
twoimo changed the base branch from stability/3-worker-refactor-and-dormant-modules to main November 16, 2026 00:44
@twoimo twoimo closed this Nov 16, 2026
@twoimo twoimo reopened this Nov 16, 2026
@twoimo
twoimo merged commit a7772c9 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant