Skip to content

chore: identical-argument scenario for the hints harness (RIUS-1727) - #276

Open
baharedr wants to merge 41 commits into
pablo/hints-harnessfrom
rius-1727-identical-args
Open

baharedr wants to merge 41 commits into
pablo/hints-harnessfrom
rius-1727-identical-args

Conversation

@baharedr

@baharedr baharedr commented Nov 12, 2026 •

Copy link
Copy Markdown

Stacked on #269 (spike/hints/). Turns the harness into something that can run against a real stack, and gives it tasks for the hint measurement's baseline.

Tasks (spike/hints/tasks.py), each with a known answer and a fixed success criterion (correct in the summary):

Task Flaw a real integration has Way out Success: the answer says
orders list_orders cursor never advances (page 2 returns its own token) get_customer_summary 4 orders, 1280.50
refund get_refund_status stuck on processing get_refund_events: rejected, card expired rejected / expired
docs search_docs returns nothing for any query lookup_index 7
identical pipeline check only: its prompt tells the model to repeat the same search lookup_index 7

Only identical asks for a repeat. The other three leave looping to the model.

Answer grading (review fix). Task runs request a JSON final answer with the same field names/types in both arms and no expected values in the response instructions. Counts and statuses are validated exactly, and monetary totals use decimal comparison. Missing/extra/duplicate fields, negated statuses, wrong numbers, and error answers cannot count as completed tasks. The scripted answers follow the same contract. Existing results from substring grading must not be mixed with this response contract without inspecting the original answers.

Three additional real-model tasks are included: polling (a stuck refund status), retry (an unavailable invoice service with a reporting-replica alternative), and cursor (repeating pagination without the summary tool).

Models.

  • MODEL=scripted: the task's fixed script, with LLM spans recorded by hand. --no-transcript records them without messages.
  • MODEL=stub: serves the same script as the Messages API on localhost, so the real Anthropic client and its instrumentation record the spans a paid run would, at no cost.
  • Any other value is a real model (--max-turns, default 12).

Run shape. Each task:

  • emits one TOOL span per call, with its arguments captured;
  • turns partial_spans on, because the writer only addresses a run whose root is stored while open;
  • sets the answer as the root's output.

Faults for the coverage checks: --duplicate-export, --late-after/--late-by, --regular (polling), --session + --runs.

The default search scenario and --auto-hint behave as before, except that tool calls are now TOOL spans and --auto-hint counts the task's own loop tool.

Driven by argus-core's TestHintMeasurement_RealPythonSDK (pipeline) and TestHintMeasurement_ControlTrial (control-only trial; paid models only with ARGUS_HINT_TRIAL_PAID=1), glassflow/argus-core#745.

dependabot Bot and others added 30 commits November 6, 2026 12:01
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0 to 4.6.1 (#266)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.1.0 (#267)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…to 1.45.2 (#270)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.3.0 (#272)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… 0.1.69 (#274)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
`--scenario identical` repeats search_docs with the same query, each call a
TOOL span with its arguments captured, sequential and successful, while the
run stays open: the loop CF-01's real detector needs. The scenario turns
partial spans on, since the writer only addresses a run whose root is stored
while open.

MODEL=scripted replaces the model with a fixed script and no API call: it
searches until a hint arrives, then looks the page up and answers, or gives
up after --hold seconds. Its model calls are LLM spans with the conversation
captured (none with --no-transcript), so the measurement can hold them
against the tool spans.

For the measurement's coverage checks: --session and --runs (a session keeps
its arm across runs), --duplicate-export (every batch twice), --late-after/
--late-by (batches held back), --regular (a polling loop). Each run prints
its own summary line, with its session and trace id.
The measurement tells a run that ended with an answer from one that ended
without by the root span's output; without it every harness run read as
ended_no_output.
…(RIUS-1850)

spike/hints/tasks.py adds three tasks with a known answer and a fixed success
criterion, each with a tool flaw a real integration has and a second tool
that leads to the answer:

- orders: the order listing's cursor never advances (page 2 hands back its
  own token); the monthly summary has the answer (4 orders, 1280.50).
- refund: the refund status is stuck on processing; the processor's event
  history shows it was rejected (card expired).
- docs: search returns nothing for any query; the legacy index has the page
  (7 retries). A model usually rephrases, so this shows how often a natural
  loop does not meet CF-01.

The prompts say nothing about retrying. `identical` moves into the same
table, unchanged, and stays the pipeline check: its prompt does ask for the
repeat.

MODEL=stub serves a task's script as the Messages API on localhost, so the
real Anthropic client and its instrumentation record the LLM spans a paid
run would, at no cost. The scripted model now plays any task. Every task
turns partial spans on (the writer needs the open root stored), and
--auto-hint counts the task's own loop tool. Summaries carry tool call
counts per tool, and --max-turns bounds a real model's run.
input_tokens (uncached and cache writes), cache_read_tokens and output_tokens
per run, so a paid trial reports what it spent.
…ive checks

For the scripted and stub models: --loop-calls N gives up after N loop calls
and --idle S keeps the run open S seconds before answering, so a loop can
stop while its run is still open. --drop-call K never exports the K-th span
of the loop tool, nor its snapshot (a lost call). --fail-first K and
--fail-every K give loop calls an error status (retries, which
exclude_retries leaves out). --no-hints starts the SDK without hints, so a
treatment decision is never handed over.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.3.5 (#280)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…1 to 4.7.6 (#281)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
PabloPardoGarcia and others added 11 commits November 17, 2026 09:57
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ng, retry and a stuck cursor

The orders, refund and docs tasks each offer an obvious second tool, and a
real model takes it before it repeats a call three times, so steering never
fires. polling asks for a refund's final state while its status stays
processing; retry's invoice service always answers 'temporarily unavailable,
retry'; cursor is the orders task without the monthly summary. None tells the
model to repeat.
…nd the hint receipt times

refund_case asks for a refund's final state and then for what follows from it:
the policy for the rejection reason and the customer's other refunds, each a
call that needs an earlier result. A model that polls the stuck status still
has requests to make after a tool_loop decision, so a hint can reach one.

The harness logs, with millisecond UTC times, each hint the SDK accepts
(on_hint) and injects, and the SDK's own drop and prune lines for rius.hints.
…and scripted late hints and relapses

- refund_record: a genuine loop with a reachable answer; the processor's
  events are final and say they are the system of record.
- export_poll: legitimate polling; the job is done 40 seconds after the first
  check, and only the status says how many rows it exported.
- refund_batch: three refunds with the same stuck status, for recurrence.
- --heartbeat: the ping interval (15 is the SDK's own default).
- --switch-after/--switch-for: the script changes approach after N loop
  calls, so a hint that arrives then is a late one.
- --relapse: the script loops N more calls after the first hint.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants