Repository navigation
fix(sleep): four evidence-integrity fixes from #310 - #392
Merged
Merged
Conversation
recall_similar blocked tonight's held-out tasks by id only. Ids hash the project with the intent and the recall archive is shared across projects, so another project's copy of tonight's val task was recalled into training under a different id. The leak check in _split was id-based too, so the gate certified the result: in a two-project cycle the verdict flipped from reject to accept_new_best with holdout_leaked=False. - task_content_key(): normalized intent plus context excerpt - recall_similar(exclude_tasks=...) blocks archived tasks whose content key matches a held-out task; dream_consolidate passes tonight's val/test tasks - _split treats a val task with a content twin in train as leaked, so the existing reject_unverified abstention applies on any route Follows up the content-hash hardening listed as a non-goal in #235. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 745807c)
…l route Tool-loop backends detect real calls (for example from the search shim's call log) and return an empty tools_called when the agent never ran the tool. The rule judge then fell back to a TOOL_CALL marker regex over the response text and passed the check anyway, undoing the verification one step later: a Claude replay whose shim log was empty scored 1.0. - _check / score_rule_judge(_with_feedback) take verified_tools; when set, only measured calls satisfy tool_called - replay_one sets it for tool tasks, which always go through attempt_with_tools; the inherited marker fallback still converts markers into calls there, so single-shot backends are unchanged - optimizer feedback for a self-reported call asks for a real call - unmeasured single-shot judging keeps the documented marker approximation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c1164b5)
The judge prompt asks for a 0..1 score, but CliBackend.judge accepted any float, including NaN and Infinity (json.loads parses both) and replies on another scale. On the default mixed gate metric, one out-of-range soft score let a candidate with flat hard accuracy and a 1.0 -> 0.0 regression read as 0.625 -> 0.750 and be accepted. Scores must now be finite and within [0, 1]; anything else fails closed as judge-score-out-of-range with the raw value in the rationale. Clamping was rejected because it would turn an 8/10 reply into a perfect 1.0. Boolean scores fall through to judge-parse-failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6de589a)
The learned block is read back one "- " bullet per item, but set_learned
wrote items verbatim. A multi-line item lost every continuation line the
next time any edit was applied; a "- X" add slipped past the duplicate check
for an existing "X"; and lstrip('- ') removed leading dashes such as
"--force". Losing adopted text changes what the gate scores: a helpful edit
was rejected (0.5 vs 0.5) because applying it silently deleted an adopted
rule held on a continuation line.
- _learned_item(): one item is one line (whitespace collapsed) with exactly
one leading bullet marker removed; used for add, replace and set_learned
- current_learned_lines joins continuation lines of blocks written by
earlier versions onto their item instead of dropping them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e3c5c56)
Contributor
|
Thanks so much. Appreciate it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the four evidence-integrity fixes out of #310 by Bogdan (Dan) Baciu (@bogdanbaciu21) so they can land now. The fifth change in #310, the harvest marker that also drops ordinary sessions quoting the grader prompt, stays in #310 for follow-up.
The four commits are cherry-picked unchanged from #310 (
-xrecords each origin) and keep Dan as the author:TOOL_CALL:marker. When the tool loop measured no call, the rule judge fell back to a regex over the response text and passed anyway.apply_editsround-trips.One maintainer commit on top:
ANTHROPIC_API_KEYprobes the CLI for--safe-mode; otherwise it refuses. The tool-route regression test now sets a test key, so it exercises the measured tool route rather than the isolation guard.Verification on current
main(ccfdcdf7ab50):main: 4 failures in the tool-evidence tests, 7 in the judge-contract tests and 5 in the round-trip tests; the recall-twin test cannot import onmain.Refs #310.