[Bogdan (Dan) Baciu (bogdanbaciu21)]

What Problem This Solves

The nightly gate accepts a candidate when it scores strictly higher than the current skill on tonight's validation tasks. In the documented deployment recipe that slice is one to seven tasks (mean 3.3) with one replay each, so a rule that damages the agent can still come out ahead on it.

I measured how often. Six fixed rules, each with a measured held-out effect, were proposed to the shipped nightly step in 100 nightly contexts (20 seeds, 5 nights) of the SearchQA recipe from docs/sleep/RESULTS.md. Three of the rules cost between 16 and 49 held-out points. The shipped gate accepted them 24 times out of 300.

Proof

All runs use public data and the shipped code path: the benchmark split materialized from the released ID manifest, the in-tree evaluator, and dream_consolidate called as the nightly cycle calls it. The harness is in this PR as skillopt_sleep/experiments/gate_study.py; Reproduce It Yourself gives one command per result. Two scenarios were measured live: SearchQA with gpt-5.4-nano (sections 1 to 3) and the checks in section 4 on a second target model and a second benchmark. The feature-on arms of sections 1 and 2 ran on this branch before its rebase (tree 84765e6; the rebase moved one CHANGELOG entry and changed nothing else). The feature-off arm of section 1 ran on an earlier head of this branch and the feature-off arm of section 2 on upstream main ccfdcdf; with the key unset both take the shipped path.

Target gpt-5.4-nano, optimizer gpt-5.5, gate metric mixed, seeds 101 to 120, held-out exact match on the first 400 test items. Arms share every model draw they have in common. Intervals are percentile bootstraps over seeds (10,000 resamples); the paired test is from python -m skillopt_sleep.evalkit.

1. Fault injection through the nightly step, feature off versus feature on. Each rule is proposed in place of the optimizer's reflection; the starting skill is the shipped SearchQA initial skill (held-out exact match 50.2); one rollout, recall_k=10.

Rule (measured held-out effect)	Acceptance, off	Acceptance, on
H4 add the category in parentheses (-49.2)	6 of 100	0 of 100
H2 answer in a full sentence (-28.5)	4 of 100	0 of 100
H1 answer with the first document title (-16.2)	14 of 100	1 of 100
H3 list alternatives (+3.5)	see near-null row
H5 prefer the formal name (+0.5)	see near-null row
H6 placebo, be careful and precise (+3.5)	see near-null row

Group	Acceptance, off	Acceptance, on	Difference (95% interval)	McNemar exact
Rules costing more than 15 points (300 decisions)	24 (8.0%)	1 (0.3%)	-7.7 points (-11.0 to -4.7)	p < 0.001, 23 versus 0 discordant
Near-null rules (300 decisions)	83 (27.7%)	79 (26.3%)	-1.3 points (-2.7 to -0.3)

Target calls per decision: 20.6 off, 21.8 on for the costly rules.

2. Paired end-to-end run with real evolution, feature off versus feature on. The full recipe (nights=5, per_night=10, dream_rollouts=5, dream_factor=2, recall_k=10), optimizer proposals untouched.

Outcome	Off	On
Held-out exact match after nights 0 to 5	50.2, 53.6, 58.5, 60.1, 63.8, 64.5	50.2, 52.9, 57.8, 59.4, 63.9, 64.7
Seeds with an identical final skill		19 of 20
Nights with an accepted edit (acceptance)	33 of 100	34 of 100
Mean held-out exact match over nights 1 to 5		-0.36 points (-1.09 to 0.00)
Held-out exact match after night 5		+0.15 points (0.00 to +0.45)
Target calls per night	218	223 (+2%)
Target tokens per night (input plus output)	286,030	291,727 (+2%)

On the one seed that differs, the check blocked a first-night candidate that an audit scored at +13.5 held-out points. That seed accepted edits on nights 4 and 5 instead and finished at 66.8 against 63.8. This is the check's measured false-veto cost on real proposals: 1 of 33.

3. Supplementary: the decision recomputed on stored samples (independent draws, 18 seeds). This table comes from stored per-task scores of the same study and is not reproducible from this PR alone; it is included because it separates real optimizer candidates from constructed rules. For 89 of the 90 audited optimizer candidates (the remaining one, a first-night candidate that measured -6.2, fits neither real row) and for the six rules in 90 contexts, fresh validation and training samples were drawn under the current skill and the candidate, and the decision was computed with and without the check.

Candidates	Held-out effect	Acceptance, off	Acceptance, on
Real, proposed from the initial skill (41)	+5 to +16.5	39.0%	39.0%
Real, later nights (48)	mean +1.5	35.4%	35.4%
Constructed, cost more than 15 points (270)	-16 to -49	10.4%	0.7%
Constructed, near-null (270)	-2 to +4	29.3%	28.1%

4. A second target model and a second benchmark (run on this head, 20d8114). The design for the second benchmark was fixed before its first call. The second-model run was added after the second benchmark's rule effects were seen, for the reason given below, and is labelled as such. Both are smaller than section 1.

4a. Second target model, gpt-5.4-mini on SearchQA, fault injection with the same six rules. Rule effects on this model (starting skill 72.8 held out; H1 costs 26.2 points and H4 72.8) were measured earlier on the first 400 test items and are reused, not re-measured. I first ran 6 seeds (30 contexts). The shipped gate accepted only 2 costly proposals there, too few to tell anything, so I extended to 20 seeds (100 contexts). The extension, its size and its stopping point were written down before the added runs, but it was decided after looking at the 6-seed result, so the 20-seed column is not a pre-planned test and both are shown.

Group	6 seeds: off, on	6 seeds: difference (95% interval)	20 seeds: off, on	20 seeds: difference (95% interval)	20 seeds: discordant
Rules costing more than 15 points, H1 and H4	2, 0 of 60	-3.3 points (-6.7 to 0.0)	8, 0 of 200	-4.0 points (-7.0 to -1.5)	8 versus 0
Moderate, H3 at -10.5	2, 1 of 30	-3.3 points (-10.0 to 0.0)	14, 7 of 100	-7.0 points (-12.0 to -2.0)	7 versus 0
Near-null, H2, H5, H6	10, 8 of 90	-2.2 points (-4.4 to 0.0)	45, 37 of 300	-2.7 points (-5.0 to -0.7)	8 versus 0

At 20 seeds the check removed all 8 costly accepts (4 of H1, 4 of H4; 6 of the 8 fell in the 14 added seeds) and the interval excludes zero. At the 6 seeds first run it did not. It also removed 8 of 45 near-null accepts, which is its cost on this model. Target calls per decision at 20 seeds: 20.6 off, 21.2 on for the costly rules.

4b. Second benchmark, LiveMathematicianBench with gpt-5.4-nano. Nightly pool: the 52 items of the released train and val manifests, five nights of ten; held out: the 125 test items; choices shuffled per item with the dataloader's rule. Six rules written before any call were measured first (starting skill 26.4 held out):

Rule (measured held-out effect, n = 125)	Acceptance, off	Acceptance, on
M5 restate the hypotheses (-7.2)	13 of 60	13 of 60
M1 always the last option (-4.0)	17 of 60	14 of 60
M6 placebo, be careful and precise (-4.0)	15 of 60	14 of 60
M2 the weakest statement (-1.6)	9 of 60	7 of 60
M3 no step-by-step reasoning (+0.8)	8 of 60	6 of 60
M4 the longer option (+0.8)	16 of 60	13 of 60

No rule costs more than 15 points here, so this benchmark cannot test whether the check rejects costly rules; that was the outcome I had registered as uninformative for the main claim. The target answers 26 percent on five-option questions, so a rule cannot both cost 15 points and beat the current skill on a validation slice, and I did not add stronger rules after seeing these effects. What the run does show is a cost: over the five near-null rules the check lowered acceptance from 65 to 54 of 300 (-3.7 points, -6.3 to -1.3; 11 versus 0 discordant), and it left the one moderately harmful rule untouched (13 to 13). Target calls per decision: 21.0 off, 23.8 on.

4c. Paired multi-night run with the optimizer proposing, LiveMathematicianBench (gpt-5.5 optimizer, 5 seeds, 5 nights, full recipe).

Outcome	Off	On
Held-out accuracy after nights 0 to 5	26.4, 26.2, 26.9, 26.9, 24.6, 24.6	identical
Nights with an accepted edit (acceptance)	5 of 25	5 of 25
Seeds with an identical trajectory		5 of 5
Held-out difference, mean over nights 1 to 5 and final		0.00 and 0.00
Target calls per night	217	219 (+1%)

The check ran on every accepted candidate and blocked none, so the two arms are the same run. Three of five seeds accepted nothing. A counter-example belongs here: on seed 101, night 4, the gate accepted an edit after which held-out accuracy fell from 29.6 to 20.0. The check did not stop it. On that night the current skill solved 1 of the 18 training tasks and the candidate solved the same 1 (no task worse, none better), so there was no drop to see. The check detects damage as tasks the current skill handles and the candidate breaks; when the current skill handles almost none, it has nothing to measure. Seed 105 shows the two-condition rule from the other side: on night 1 the candidate's training mean fell from 0.14 to 0.00, beyond the margin, but only one task got worse, so it was not blocked, and held-out accuracy moved from 26.4 to 25.6.

Why This Change Was Made

gate_train_check (default false) adds a condition to an accept. A candidate the validation gate would accept is also replayed on tonight's real training tasks (recalled tasks included, dream variants not). It is rejected when both hold: its mean gate score there is lower than the current documents' by more than gate_train_margin (default 0.1), and at least two more of those tasks got worse than got better.

Design choices that matter for review:

• It only rejects. Training tasks are not held out, since reflection drew on tonight's failures, so a higher score on them cannot rule out overfitting and is never used to accept anything. A clearly lower score on them, in particular breaking tasks the current documents already handle, is not explained by overfitting.
• It is baseline-relative and aggregate. The candidate is compared with the documents it would replace on identical tasks, both arms and every task-level change are kept in the evidence, and one task flipping can never reject: it takes a mean drop beyond the margin and a net of two regressed tasks.
• A skill candidate is compared with the night's existing training replay; a memory candidate with the replay that already follows an accepted skill. The only added cost is one replay of the training tasks per candidate that passes validation.
• A task with a missing or malformed score counts as worse and a non-finite mean rejects. The check is skipped when the gate is off and when the validation slice is not disjoint from training.
• With the key unset, calls, records, evidence events and report.md are what they were: the new record keys are written only when the check is enabled.

Non-goals: this does not make the gate accept more, does not replace held-out validation, and does not detect harm that tonight's training tasks cannot show.

Project Fit

• Improving SkillOpt-Sleep with evidence: the validation gate is the documented safety argument of the nightly loop, and on the SearchQA recipe this narrows a measured hole in it at about 2 percent more target calls.
• Robustness across scenarios, reported as measured: the effect is large on SearchQA with gpt-5.4-nano, is also seen on a second model once that run was extended to 20 seeds, and is absent on a second benchmark where the agent starts near chance. The proof shows each case, including one harmful accept the check missed.

User Impact

Opt-in and default-unchanged. A user who sets "gate_train_check": true gets a nightly gate that, on the measured recipe, stopped accepting rules that cost 16 to 49 held-out points (24 of 300 down to 1 of 300) while leaving the real learning path unchanged on 19 of 20 seeds. report.md shows the two means, the counts of tasks that got worse and better, and whether the candidate was rejected for it; the JSON outputs and the evidence log keep the per-task comparison.

Academic Support

• Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, Stefano Soatto, "Positive-Congruent Training: Towards Regression-Free Model Updates", CVPR 2021, arXiv:2011.09161. Defines the negative flip, a sample the old version got right and the new one gets wrong, and argues for limiting it separately from the average error. The check's second condition, at least two more tasks worse than better, is a negative-flip count on tonight's training tasks.
• Yuqing Xie, Yi-an Lai, Yuanjun Xiong, Yi Zhang, Stefano Soatto, "Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates", ACL 2021, arXiv:2105.03048. Measures negative flips across NLP model updates and finds them prevalent across the GLUE tasks; a candidate that wins a three-task validation slice can still carry such regressions.
• David Lopez-Paz, Marc'Aurelio Ranzato, "Gradient Episodic Memory for Continual Learning", NeurIPS 2017, arXiv:1706.08840. Allows an update only if the loss on stored examples does not increase; this check is the reject-only analogue for a text edit, with tonight's training tasks as the stored examples.
• Romain Laroche, Paul Trichelair, Remi Tachet des Combes, "Safe Policy Improvement with Baseline Bootstrapping", ICML 2019, arXiv:1712.06924. Falls back to the baseline where the data do not support a change; the same baseline-relative stance, and the reason the check needs more than one task to act.
• Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth, "Generalization in Adaptive Data Analysis and Holdout Reuse", 2015, arXiv:1506.02629. Data that an adaptive procedure has already drawn on cannot certify an improvement, which is why the training side is used only to reject.
• Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, Michael Zeng, "Automatic Prompt Optimization with 'Gradient Descent' and Beam Search", EMNLP 2023, arXiv:2305.03495. Selects among candidate prompts with a bandit procedure because scoring each candidate on all data is too expensive; a validation slice of about three tasks is the small-sample end of that selection problem.
• A limit from prior work: Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul, "Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting", 2026, arXiv:2610.12601. Reports that a reflective optimizer can write training answers into the prompt. Such a candidate keeps or raises its training score, so this check cannot catch it; it catches breakage, not overfit edits.

Testing

New tests in tests/test_sleep_gate_train_check.py (62 cases):

• Default: no extra replay, identical calls, and exactly the pre-existing record keys and evidence events with the key unset, false, or null.
• Behaviour: a candidate that breaks its own training tasks is rejected with both arms kept; a helpful candidate is accepted; nothing validation rejected is ever accepted or replayed; a candidate already blocked by gate_no_regression is not replayed.
• The rule: the margin is the tolerated absolute drop; a drop of exactly the margin is tolerated at every baseline (the float boundary); margin zero; one task flipping never rejects; improved tasks offset regressed ones; a malformed score and a task missing from the replay fail closed.
• Scope: recalled tasks are checked, dream variants are in neither arm, gate off and a non-disjoint slice skip it, dream rollouts with parallel workers still run it.
• Skill then memory: the memory candidate is compared with the post-skill replay; a blocked skill does not become the memory baseline.
• Dual backend: the check replays on the target model only and the optimizer model is not called.
• Settings and records: non-boolean flag and invalid margin stop before any model call; evidence keeps both arms; report wording for rejected, lower-but-not-enough and clean; both nightly call sites forward the settings; a config file is read; a null margin means the default.

The existing fan-out forwarding test in tests/test_sleep_engine.py also asserts the two settings.

tests/test_sleep_gate_study.py (6 cases) runs the study harness offline on the committed fixture: the fixture's rule effects, the shipped gate accepting the harmful fixture rule in 1 of 16 contexts and the check in 0 of 16, the helpful rule accepted equally in both arms, the cost of the extra training replay, the refusal to call a model without --live or --offline, and the report arithmetic on known rows.

Line and branch coverage of the source lines this PR adds in skillopt_sleep/consolidate.py and skillopt_sleep/cycle.py, measured with pytest --cov --cov-branch on the two test files above plus tests/test_sleep_engine.py: 85 of 85 lines and 38 of 38 branches.

Complete suite, this branch at 20d81145f10d44c284e7889042a60502faf5ac6e and unmodified upstream main 45f65ca on the same runners. Each job checked out the exact commit and asserted its SHA. These are contributor-run results, not official CI.

Platform	Python	This branch	Upstream main
Linux (Ubuntu)	3.10	2272 passed, 0 failed, 24 skipped, 4 xfailed	2204 passed, 0 failed
Linux (Ubuntu)	3.11	2272 passed, 0 failed, 24 skipped, 4 xfailed	2204 passed, 0 failed
Linux (Ubuntu)	3.12	2272 passed, 0 failed, 24 skipped, 4 xfailed	2204 passed, 0 failed
macOS arm64	3.12	2270 passed, 2 failed, 24 skipped	2202 passed, 2 failed
Windows	3.12	2197 passed, 24 failed, 80 skipped	2129 passed, 24 failed, 80 skipped

The branch passes exactly 68 more tests than main on every runner, which is the 62 and 6 new cases. The failing test ids are identical on the branch and on main: 2 on macOS (test_concurrent_adoption_cleanly_refuses_one_writer and test_separate_nights_share_the_same_live_target_lock in tests/test_sleep_adopt_skill_subset.py) and 19 ids on Windows (24 failures counting subtests). None is unique to this branch and none touches code this change modifies; they are reported, not fixed, to keep the PR scoped.

On Linux 3.12 locally: mkdocs build --strict exits 0, ruff check on the changed source and test files is clean, and git diff --check is clean. The live runs in Proof sections 1 to 3 were made before the rebase from ccfdcdf onto 45f65ca; section 4 ran on the head above.

Limitations & Negative Results

• It wrongly blocked one good candidate in 20 seeds (1 of 33 accepted candidates), delaying that seed's first accepted edit by three nights.
• The harmful rules are constructed, not optimizer-proposed. In these runs the optimizer almost never proposed a clearly harmful edit: of 90 audited real candidates, 4 measured below -2 held-out points, none below -7.
• It catches large damage. A rule that costs a few points moves the training mean by less than the margin and passes; on the second model the rule costing 10.5 points was accepted 14 of 100 times without the check and 7 of 100 with it, and on LiveMathematicianBench the rule costing 7.2 points was accepted 13 of 60 times either way.
• Evidence beyond one benchmark and one model is thinner than section 1. On a second target model the check removed all 8 costly accepts in 200 decisions, but that 20-seed figure comes from an extension decided after the first 6 seeds had shown 2 to 0 with an interval including zero (section 4a). On LiveMathematicianBench no test rule was costly, the check lowered acceptance of near-null rules by 3.7 points, and in the multi-night run it changed nothing (sections 4b and 4c).
• It also blocks some harmless edits. Near-null rules lost 1.3 points of acceptance on SearchQA with gpt-5.4-nano, 2.7 points with gpt-5.4-mini and 3.7 points on LiveMathematicianBench.
• The check has no power when the current skill solves few of tonight's training tasks. On LiveMathematicianBench the gate accepted an edit that cost 9.6 held-out points and the check saw no drop, because the skill solved 1 of 18 training tasks before and after (section 4c).
• The two-task condition was added after a first version (mean drop only, at least four training tasks) vetoed 2 of 41 real candidates in the decision study. All numbers above are for the final rule; with the first rule the fault-injection result was 0 of 300 and 78 of 300.
• The margin applies per candidate, so a skill edit and a memory edit accepted on the same night can each use it.
• In per-skill fan-out a group's check appears in the evidence log and JSON, not as its own line in report.md.
• One mechanism tested in the same study is not proposed because it did not help: probing each task with two rollouts before the full five (40 percent fewer calls, learning curve -1.36 points, -2.88 to -0.01).
• Section 3 of Proof rests on stored per-task scores that are not part of this PR, so a reviewer cannot re-derive that table; sections 1, 2 and 4 are reproducible with the harness.

Reproduce It Yourself

Deterministic tests, and the study harness offline (no key, no network):

python -m pytest tests/test_sleep_gate_train_check.py tests/test_sleep_gate_study.py tests/test_sleep_engine.py -q
python -m pytest -q
python -m skillopt_sleep.experiments.gate_study effects --offline
python -m skillopt_sleep.experiments.gate_study faults --offline --arm off
python -m skillopt_sleep.experiments.gate_study faults --offline --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study report --offline

Live numbers. They need the searchqa extra, the openai package and any OpenAI-compatible endpoint (OPENAI_API_KEY, optional OPENAI_BASE_URL). Each command needs --live and prints the number of model calls it expects before the first one. Every model call is kept in a local SQLite store, so runs resume and the second arm reuses the first arm's draws.

Proof section 1 (fault injection, SearchQA):

python -m pip install -e ".[searchqa]" openai
python scripts/materialize_searchqa.py
python -m skillopt_sleep.experiments.gate_study effects --live
python -m skillopt_sleep.experiments.gate_study faults --live --arm off
python -m skillopt_sleep.experiments.gate_study faults --live --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study report

Proof section 2 (multi-night run with the optimizer proposing):

python -m skillopt_sleep.experiments.gate_study nights --live --arm base
python -m skillopt_sleep.experiments.gate_study nights --live --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study report

Proof section 4 (second target model, second benchmark):

python -m skillopt_sleep.experiments.gate_study faults --live --model gpt-5.4-mini --arm off
python -m skillopt_sleep.experiments.gate_study faults --live --model gpt-5.4-mini --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study materialize --benchmark livemath
python -m skillopt_sleep.experiments.gate_study effects --live --benchmark livemath --test-n 125
python -m skillopt_sleep.experiments.gate_study faults --live --benchmark livemath --seeds 12 --arm off
python -m skillopt_sleep.experiments.gate_study faults --live --benchmark livemath --seeds 12 --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study nights --live --benchmark livemath --seeds 5 --test-n 125 --arm base
python -m skillopt_sleep.experiments.gate_study nights --live --benchmark livemath --seeds 5 --test-n 125 --arm check --feature '{"gate_train_check": true}'
python -m skillopt_sleep.experiments.gate_study report --benchmark livemath

On SearchQA, effects is 2,800 target calls and faults --arm off about 4,000 per 20 seeds, of about 1,400 input tokens each; the feature-on arm adds a few hundred. A provider that names models with a prefix takes --model and --optimizer accordingly.