Repository navigation
Conversation
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…0 to 4.6.1 (#266) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.1.0 (#267) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…to 1.45.2 (#270) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.3.0 (#272) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… 0.1.69 (#274) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
`--scenario identical` repeats search_docs with the same query, each call a TOOL span with its arguments captured, sequential and successful, while the run stays open: the loop CF-01's real detector needs. The scenario turns partial spans on, since the writer only addresses a run whose root is stored while open. MODEL=scripted replaces the model with a fixed script and no API call: it searches until a hint arrives, then looks the page up and answers, or gives up after --hold seconds. Its model calls are LLM spans with the conversation captured (none with --no-transcript), so the measurement can hold them against the tool spans. For the measurement's coverage checks: --session and --runs (a session keeps its arm across runs), --duplicate-export (every batch twice), --late-after/ --late-by (batches held back), --regular (a polling loop). Each run prints its own summary line, with its session and trace id.
The measurement tells a run that ended with an answer from one that ended without by the root span's output; without it every harness run read as ended_no_output.
…(RIUS-1850) spike/hints/tasks.py adds three tasks with a known answer and a fixed success criterion, each with a tool flaw a real integration has and a second tool that leads to the answer: - orders: the order listing's cursor never advances (page 2 hands back its own token); the monthly summary has the answer (4 orders, 1280.50). - refund: the refund status is stuck on processing; the processor's event history shows it was rejected (card expired). - docs: search returns nothing for any query; the legacy index has the page (7 retries). A model usually rephrases, so this shows how often a natural loop does not meet CF-01. The prompts say nothing about retrying. `identical` moves into the same table, unchanged, and stays the pipeline check: its prompt does ask for the repeat. MODEL=stub serves a task's script as the Messages API on localhost, so the real Anthropic client and its instrumentation record the LLM spans a paid run would, at no cost. The scripted model now plays any task. Every task turns partial spans on (the writer needs the open root stored), and --auto-hint counts the task's own loop tool. Summaries carry tool call counts per tool, and --max-turns bounds a real model's run.
input_tokens (uncached and cache writes), cache_read_tokens and output_tokens per run, so a paid trial reports what it spent.
…ive checks For the scripted and stub models: --loop-calls N gives up after N loop calls and --idle S keeps the run open S seconds before answering, so a loop can stop while its run is still open. --drop-call K never exports the K-th span of the loop tool, nor its snapshot (a lost call). --fail-first K and --fail-every K give loop calls an error status (retries, which exclude_retries leaves out). --no-hints starts the SDK without hints, so a treatment decision is never handed over.
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…to 3.3.5 (#280) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…1 to 4.7.6 (#281) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
… the newest-instrumentor checks (#294)
…y, and cover the orphan and native cases (#295)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ng, retry and a stuck cursor The orders, refund and docs tasks each offer an obvious second tool, and a real model takes it before it repeats a call three times, so steering never fires. polling asks for a refund's final state while its status stays processing; retry's invoice service always answers 'temporarily unavailable, retry'; cursor is the orders task without the monthly summary. None tells the model to repeat.
# Conflicts: # src/rius/_semconv_generated.py
…nd the hint receipt times refund_case asks for a refund's final state and then for what follows from it: the policy for the rejection reason and the customer's other refunds, each a call that needs an earlier result. A model that polls the stuck status still has requests to make after a tool_loop decision, so a hint can reach one. The harness logs, with millisecond UTC times, each hint the SDK accepts (on_hint) and injects, and the SDK's own drop and prune lines for rius.hints.
…and scripted late hints and relapses - refund_record: a genuine loop with a reachable answer; the processor's events are final and say they are the system of record. - export_poll: legitimate polling; the job is done 40 seconds after the first check, and only the status says how many rows it exported. - refund_batch: three refunds with the same stuck status, for recurrence. - --heartbeat: the ping interval (15 is the SDK's own default). - --switch-after/--switch-for: the script changes approach after N loop calls, so a hint that arrives then is a late one. - --relapse: the script loops N more calls after the first hint.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #269 (
spike/hints/). Turns the harness into something that can run against a real stack, and gives it tasks for the hint measurement's baseline.Tasks (
spike/hints/tasks.py), each with a known answer and a fixed success criterion (correctin the summary):orderslist_orderscursor never advances (page 2 returns its own token)get_customer_summaryrefundget_refund_statusstuck onprocessingget_refund_events: rejected, card expireddocssearch_docsreturns nothing for any querylookup_indexidenticallookup_indexOnly
identicalasks for a repeat. The other three leave looping to the model.Answer grading (review fix). Task runs request a JSON final answer with the same field names/types in both arms and no expected values in the response instructions. Counts and statuses are validated exactly, and monetary totals use decimal comparison. Missing/extra/duplicate fields, negated statuses, wrong numbers, and error answers cannot count as completed tasks. The scripted answers follow the same contract. Existing results from substring grading must not be mixed with this response contract without inspecting the original answers.
Three additional real-model tasks are included:
polling(a stuck refund status),retry(an unavailable invoice service with a reporting-replica alternative), andcursor(repeating pagination without the summary tool).Models.
MODEL=scripted: the task's fixed script, with LLM spans recorded by hand.--no-transcriptrecords them without messages.MODEL=stub: serves the same script as the Messages API on localhost, so the real Anthropic client and its instrumentation record the spans a paid run would, at no cost.--max-turns, default 12).Run shape. Each task:
partial_spanson, because the writer only addresses a run whose root is stored while open;Faults for the coverage checks:
--duplicate-export,--late-after/--late-by,--regular(polling),--session+--runs.The default
searchscenario and--auto-hintbehave as before, except that tool calls are now TOOL spans and--auto-hintcounts the task's own loop tool.Driven by argus-core's
TestHintMeasurement_RealPythonSDK(pipeline) andTestHintMeasurement_ControlTrial(control-only trial; paid models only withARGUS_HINT_TRIAL_PAID=1), glassflow/argus-core#745.