Repository navigation
feat(reports): mark cross-model turn comparisons in experiment reports - #262
NopeCharon wants to merge 2 commits into
Conversation
Turn counts depend on the model: one model chains a whole phase into a single Bash call, another sends one probe per call, so the same task can take twice the turns on a different model. The experiment report's Aggregate Metrics table compared Assistant Turns across variants with a Welch p-value even when the variants ran different models, so a model comparison (or an arm-vs-arm run on mixed models) printed a turn delta that was mostly model style, not variant cost. - VariantResult gains `models_used`: the distinct `model_used` values across the folded replicates. Defaults to [] so an older experiment.json still loads. - The Aggregate Metrics table (experiment.md and experiment.html) gains a Model row. When the variants ran more than one model, the turns row reads "Assistant Turns (cross-model)", has no p-value, and a shared note under the table says why. Score, duration and token rows keep their p-values. - Model ids are compared after pricing's normalization (now public as `normalize_model`), so a row that errored before the model resolved and kept the qualified route id is not a second model. Claude Code's `<synthetic>` placeholder (a CLI-made error message) is not a model either. A variant with no recorded model is unknown, never different. - Docs: the visible-turns section of the Task Definition Guide states the same-model rule, AB_EXPERIMENTS describes the Model row and the marker, and REPORT_SCHEMA lists the new field. Part of UiPath#98. The evalboard trend surfaces are not changed here.
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:262 (14 files) axis:1,2,3,4,5,6,7,8
Scope: pr:262 (14 files) axis:1,2,3,4,5,6,7,8 · branch feat/cross-model-turns (pr-262) · ece170e · 2026-11-15T23:43Z · workflow variant
Change class: complex — adds a persisted VariantResult.models_used field and changes experiment-report statistics semantics (suppresses the Assistant Turns p-value on cross-model comparisons) in both markdown and HTML renderers
PR #262 is a clean, well-contained report-only change: it causes no change to type safety, security, architecture or error handling, and no finding can change a task's score or final_status for identical agent output; but the cross-model check compares model ids that are not fully canonical, so a same-model A/B with one crashed row (config alias claude-haiku-4-5 next to the resolved claude-haiku-4-5-20251001) gets the label "cross-model" and loses its turns p-value, and the <synthetic> and normalization cleanup occurs only in the renderer, not in the persisted models_used. Bottom line: merge after you make model identity canonical at fold time and close the small test and wording gaps.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9.4 / 10 | 0 | 0 | 1 | 1 | Persisted experiment.json models_used stores raw ids (including Claude Code's sentinel and route-qualified ids); the distinct-model cleanup runs only at render time in reports/helpers.py |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 9.8 / 10 | 0 | 0 | 0 | 2 | Experiment report tests skip the negative assertions (note hidden without turn data, turns p-value kept for same-model runs), mainly in the HTML path |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 9.9 / 10 | 0 | 0 | 0 | 1 | The cross-model note talks about a p-value even when the table has no p-value column |
| 8. Evaluation Harness Quality | 9.5 / 10 | 0 | 0 | 1 | 0 | Cross-model detection treats a dated snapshot id and its alias as two models, so a same-model A/B loses its turns p-value and is labelled cross-model |
Overall Score: 9.8 / 10 · Weakest Axis: Code Quality & Style at 9.4 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 2 · 🔵 4 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
- [Axis 1] Persisted experiment.json models_used stores raw ids (including Claude Code's sentinel and route-qualified ids); the distinct-model cleanup runs only at render time in reports/helpers.py (
src/coder_eval/reports/helpers.py:143) — helpers.py:143 adds_SYNTHETIC_MODEL = "<synthetic>"and line 157 filters on it:models[vr.variant_id].update(normalize_model(m) for m in vr.models_used if m != _SYNTHETIC_MODEL). This is the 'hardcoded magic-string special case where a generic mechanism fits' anchor (Medium). One agent's quirk is now handled inreports/, which is supposed to work the same for every agent. The value is cleaned only when the report renders. The producer still stores it: claude_code_agent.py:408-410 (model_attr = getattr(message, "model", None)/if isinstance(model_attr, str): self.sdk_model_used = model_attr) accepts<synthetic>. The fold at orchestration/experiment.py:848 (models_used=sorted({r.result.model_used for r in reps if r.result.model_used})) then saves raw, un-normalized ids into experiment.json. So the persistedmodels_used,EvaluationResult.model_usedand the pricing pre-flight (unpriced_models) all still see<synthetic>as a model, and the cleaning is split across two layers. Fix: stop Claude Code from recording<synthetic>assdk_model_usedat the producer. Do thenormalize_modelstep once in_fold_replicates. Thenvariant_modelsbecomes a plain union and the constant can be deleted. - [Axis 8] Cross-model detection treats a dated snapshot id and its alias as two models, so a same-model A/B loses its turns p-value and is labelled cross-model (
src/coder_eval/reports/helpers.py:157) —variant_modelsnormalizes withpricing.normalize_model:models[vr.variant_id].update(normalize_model(m) for m in vr.models_used if m != _SYNTHETIC_MODEL). That function removes route, region,anthropic.and-vN:Mqualifiers, but it does not reduce a dated snapshot id to its alias. A Claude Code row that gets a reply records the API's resolved id (the local runs/ corpus has 54 rows with"model_used": "claude-haiku-4-5-20251001", and pricing.py has both"claude-haiku-4-5"and"claude-haiku-4-5-20251001"as keys). A row that crashes before any assistant message falls back in orchestrator.py:1136-1137 (if not self.result.model_used and self.task.agent is not None and self.task.agent.model: self.result.model_used = self.task.agent.model), so it records the configured alias (tasks/ hasmodel: claude-haiku-4-5). I ran this on the PR head:turns_cross_model({'a': ['claude-haiku-4-5','claude-haiku-4-5-20251001'], 'b': ['claude-haiku-4-5-20251001']})returnsTrue. The result is that a single-model experiment with one crashed row showsAssistant Turns (cross-model), drops the Welch p-value from both experiment.md (reports/experiment.py:228p = None if cross_model else ...) and experiment.html, prints the 'more than one model' note, and puts two ids in the Model row. This is the same class of bug the docstring says the normalization prevents ('a raw comparison would read it as a second model'). Fix: compare on one canonical model identity. Either resolve the alias at fold time (record the SDK-resolved id, or ignore the config fallback when another replicate gives a resolved id), or add a canonicalizer that maps a dated snapshot to its alias. Add a test next totest_route_qualified_id_is_the_same_modelwith['claude-haiku-4-5', 'claude-haiku-4-5-20251001']against['claude-haiku-4-5-20251001']. Severity: this is report-only and does not change score or final_status. Per the Axis 8 Medium anchor, the report misstates whether a cross-run comparison is valid.
Nits
- [Axis 1] Cross-model row logic is written twice in the markdown and HTML renderers; only the note text was moved to a shared helper (
src/coder_eval/reports/html.py:1443) — The PR movesCROSS_MODEL_TURNS_NOTEinto helpers 'so the markdown and HTML reports say the same thing', but the other new parts are still written out in both renderers. (1) The label: html.py:1443"Assistant Turns (cross-model)" if cross_model else "Assistant Turns"and experiment.py:224row = "| Assistant Turns (cross-model)" if cross_model else "| Assistant Turns". (2) The model cell: html.py:1397", ".join(models[vid]) or "n/a"and experiment.py:273', '.join(models[vid]) or 'n/a'. (3) The same rationale comment: html.py:1392-1393 and experiment.py:266-267 ('Turns depend on the model, so the table that shows them also shows what each variant ran on'). (4) The rule that shows the note only whencross_modelis set and turn data exists. In experiment.py the turn-data checkany(series[vid].asst_turns for vid in result.variant_ids)is also computed twice (lines 223 and 293); html.py computes it once ashas_turns. Put the turns label and model-cell formatting in helpers next to the note, or return them fromvariant_models, and computehas_turnsonce in_aggregate_metrics_lines. This also stops_experiment_aggregate_metrics(radon E(31)) from getting more branches. Low because the PR follows the existing pattern of building rows separately in each renderer. - [Axis 3] Experiment report tests skip the negative assertions (note hidden without turn data, turns p-value kept for same-model runs), mainly in the HTML path (
tests/test_experiment_reports.py:1419) —test_html_marks_the_same_rows_as_markdown(line 1419) checks two HTML cases: cross-model and same-model. It never runs the has_turns=False side of the new HTML gate at src/coder_eval/reports/html.py:1459:note = f'\n<p class="muted">{_esc(CROSS_MODEL_TURNS_NOTE)}</p>' if cross_model and has_turns else "". The markdown twin hastest_no_note_without_turn_data(line 1409) for its own gate at experiment.py:293. The HTML renderer has no equivalent. So if the HTML gate dropsand has_turns, it renders the note under a table that has no turns row, and no test fails. The no-model case is weaker: existing HTML snapshots built without models_used cover 'no Model row' only implicitly. Fix: add the no-turn-data case to the HTML test. Set everyvr.total_assistant_turns = Noneon_make_result({"a": ["gpt-5"], "b": ["claude-opus-5"]}), then assert thathtml_escape(CROSS_MODEL_TURNS_NOTE) not in outand"Assistant Turns" not in out. Also assert that_make_result({"a": [], "b": []})renders no<td>Model</td>. - [Axis 3] Test claims compatibility with old experiment.json files but never loads a record without the models_used key (
tests/test_experiment_reports.py:1404) — The docstring oftest_no_recorded_model_renders_no_model_rowsays "An experiment.json written before models_used existed renders as before." But the test builds every VariantResult with an explicitmodels_used=models[vid]set to[](via_make_result, line 1321), so the missing-key default path never runs. The PR documents this as a contract in docs/REPORT_SCHEMA.md ('[]... in anexperiment.jsonwritten before the field existed'). Fix: make the test exercise that path. Example:VariantResult.model_validate({k: v for k, v in vr.model_dump().items() if k != "models_used"})and assert.models_used == []. Or change the docstring so it describes only the case it really tests. - [Axis 7] The cross-model note talks about a p-value even when the table has no p-value column (
src/coder_eval/reports/helpers.py:133) —CROSS_MODEL_TURNS_NOTEstarts with "These turn counts come from more than one model, so the Assistant Turns row has no p-value." and ends with "Compare turns only between variants on the same model." Both renderers print it whenevercross_model and has_turnsis true. But p-values exist only whenlen(result.variant_ids) == 2(reports/experiment.py:252, reports/html.py:1375). So a 1-variant or 3+-variant experiment gets an explanation for a missing p-value in a table that has no p-value column. The testtest_mixed_models_inside_one_variant_are_cross_modelasserts this for a single-variant run. Recommendation: say 'no p-value' only when show_p_values is true, or reword the note so that it does not depend on the column (for example: 'These turn counts come from more than one model; a turn difference across models is mostly model style, not variant cost.').
What's Missing
Parallel paths:
- 🔵 The other model displays in the same run still show raw ids.
variant_modelsdrops<synthetic>and normalizes route-qualified ids, but only for experiment.md and experiment.html. The run.mdModel(s)header (reports/markdown.py:433sorted({t["model_used"] ...})), the run.md Task Details model column (markdown.py:559) and the task HTML header (html.py:348) were not updated. One run can therefore showclaude-sonnet-5, eu.anthropic.claude-sonnet-5, <synthetic>in run.md and a singleclaude-sonnet-5in experiment.md. If the cleanup moves to the producer or the fold, all of these surfaces agree. (trigger: src/coder_eval/reports/helpers.py) (restates: Axis 1: Persisted experiment.json models_used stores raw ids; distinct-model cleanup runs only at render time in reports/helpers.py) - 🔵 The comparing-models tutorial was not updated. docs/tutorials/05-comparing-models.md §3 'Read the comparison' still says experiment.md shows 'durations, and token usage side by side'. It says nothing about the new Model row, the 'Assistant Turns (cross-model)' label, or the missing turns p-value. This tutorial runs haiku-vs-sonnet, so every reader who follows it gets the new marker and the note. AB_EXPERIMENTS.md and TASK_DEFINITION_GUIDE.md were updated, but the tutorial that leads readers to them was not. (trigger: docs/AB_EXPERIMENTS.md)
Downstream consumers:
- 🔵 The cross-model check is applied only to the Assistant Turns row. The Tokens row in the same Aggregate Metrics table (reports/experiment.py:232-236 and reports/html.py:1453) keeps its Welch p-value across models. Token counts depend on each model's tokenizer and verbosity, so a token count from a gpt model and one from a claude model measure different things. The new docs say only that 'Cost and wall-clock time stay comparable across models'. The PR body says the token rows keep their p-values, but the docs do not explain why. Either mark the Tokens row as cross-model too, or add a line to the docs that says why tokens stay comparable. (trigger: src/coder_eval/reports/experiment.py)
- 🔵 The new paragraph ('Turn counts are comparable only within one model') goes inside the
expected_turnssection. Butexpected_turnsis still one budget per task, andexpected_turns_overage(result_metrics.py:197) checks every model against that same number. That check feeds run.jsonexpected_turns_overage, the task HTML badge and the evalboard turns cell colouring. The next paragraph still says to 'Pick budgets consistently across a suite', and it does not say that the budget depends on the model or how to set one per model (for example a variant-levelrun_limitsoverride). A variant on a model that sends one probe per call will be flagged as over budget for model style, which is exactly what the new paragraph warns about. (trigger: docs/TASK_DEFINITION_GUIDE.md)
Tests:
- 🔵 No golden snapshot covers the new table layout. The PR says the existing experiment.md and experiment.html snapshots are unchanged, because they are built without
models_used. So the Model row's position (first row, before Tasks Run), the markdown note's position after the Replicates/task row, and the HTML</div>{note}position are checked only by line or substring matches in TestCrossModelTurns. Add one snapshot fixture with mixed-model variants so a layout change in either renderer shows up as a diff. (trigger: tests/test_experiment_reports.py)
Nightly pipeline:
- 🔵 The PR does not say what happens to the nightly run. Every multi-variant nightly run will now write
models_usedinto experiment.json, which is the blob contract underruns/. Its experiment.md and experiment.html will also gain a Model row. Any nightly experiment that compares models will lose the Assistant Turns p-value. This is expected to be harmless: the field is additive with a[]default, and evalboard'sparseExperimentreads onlyvariant_aggregates. The PR also makespricing._normalize_modelpublic asnormalize_model, and no consumer was found in coder_eval_uipath. One sentence in the PR body would make this explicit. (trigger: src/coder_eval/models/experiment.py)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] CE069 (the next free id; tests/lint/runner.py says 'Claim 069 next') - new rule tests/lint/rules/ce069_no_model_canonicalization_in_reports.py, wired into ALL_RULES in tests/lint/runner.py next to CE066 (no_report_imports_in_core, its mirror image), with a TestCE069 class in tests/test_custom_lint.py. Invariant: the reports/ leaf layer shows model ids as it gets them. It must never canonicalize them. Violation class 1: any
from ..pricing import normalize_model(orpricing.normalize_modelattribute access) under src/coder_eval/reports/**. calculate_cost / is_priced stay allowed because they normalize internally for pricing only. Violation class 2: the Claude Code sentinel literal "" as a str Constant anywhere in src/coder_eval outside agents/claude_code_agent.py, because an agent-specific sentinel must be dropped inside its own adapter. Known blind spot: a hand-written prefix strip or a sentinel built at runtime. The module docstring must say so. Prevents: Finding 1 (helpers.py:29 imports normalize_model, helpers.py:143_SYNTHETIC_MODEL = "<synthetic>", helpers.py:157 filters and normalizes at render time while experiment.py:848 saves raw ids to experiment.json). It forces the cleanup into the producer (claude_code_agent.py:408-410) and _fold_replicates, so persisted VariantResult.models_used / EvaluationResult.model_used and the report share one canonical value. It also puts the Finding 6 fix (snapshot-vs-alias) upstream, where every consumer gets it, not only the renderer. - [ce-lint] CE070 - a @pytest.mark.lint data-invariant class in tests/test_custom_lint.py, in the same style as TestCE065PricingMirrorParity, that reads the rate table in src/coder_eval/pricing.py. For every key matching
^(?P<stem>.+)-20\d{6}$whose stem is also a key (today: claude-haiku-4-5 / -20251001, claude-sonnet-4-5 / -20250929, claude-opus-4-5 / -20251101, claude-opus-4 / -20250514), assert canonical_model_id(snapshot) == canonical_model_id(stem). canonical_model_id is the single model-identity function that the Finding 6 fix adds (normalize_model plus snapshot-to-alias folding), and _fold_replicates must call it. The check reads only the static table, so it maintains itself: a new dated snapshot whose alias is priced fails the build if it does not collapse to the alias. Prevents: Finding 6 (helpers.py:157: turns_cross_model({'a':['claude-haiku-4-5','claude-haiku-4-5-20251001'],'b':['claude-haiku-4-5-20251001']}) returns True, so a same-model A/B with one crashed row (config-alias fallback at orchestrator.py:1136-1137) shows 'Assistant Turns (cross-model)', loses the Welch p-value at reports/experiment.py:228 and html.py, and gets two ids in the Model row). It also stops the same split from coming back in any other place that compares model identity.
Harness improvements (not statically reachable):
- Replace the ad-hoc pairwise renderer twins in tests/test_experiment_reports.py (test_html_and_markdown_report_the_same_duration, test_html_marks_the_same_rows_as_markdown, test_html_renders_the_same_paired_numbers_as_markdown) with one parametrized markdown/HTML Aggregate Metrics parity harness. Use a fixture matrix of cross_model {T,F} x has_turns {T,F} x variant count {1,2,3} x models_used {populated, all-empty}. Get (row label, cells, p-value presence, note presence) from both experiment.md and experiment.html and assert that they are equal. Also assert two semantic invariants: 'note mentions a p-value => the table has a p-value column' and 'no models recorded => no Model row'. Why not static: The defects are in rendered output under specific data shapes: a missing
and has_turnsarm in a one-line conditional, and note text that does not match the column set. Coverage.py branch coverage does not track the arms of a ternary inside one line. A lint that intersects label literals across experiment.py and html.py would miss the case because the markdown literal has a '| ' prefix ('| Assistant Turns (cross-model)' vs 'Assistant Turns (cross-model)'). CLAUDE.md also prefers a shared helper over a guard for duplication. So the gap can only be closed by rendering both outputs and diffing them. Prevents: Finding 3 (html.py:1459 gate with no has_turns=False test, no-Model-row case covered only implicitly), Finding 5 (CROSS_MODEL_TURNS_NOTE says 'no p-value' in 1- and 3+-variant tables that have no p-value column; helpers.py:133), and Finding 2 (label / model-cell / has_turns logic duplicated at html.py:1443/1397 vs experiment.py:224/273/223/293, which can drift without notice). - Add a frozen golden experiment.json under tests/fixtures/, written before the current persisted fields existed (no models_used key). Add one test that loads it through ExperimentResult.model_validate_json, renders both experiment.md and experiment.html, and asserts that the defaults (models_used == []) apply and that no Model row or cross-model mark appears. Each later persisted VariantResult/ExperimentResult field then gets old-record coverage from this fixture, so no hand-written dict is needed per test. Why not static: To check that an old record still loads and renders, the test needs a real artifact persisted before the field existed. A docstring that claims compatibility cannot be checked against the code path it says it covers. No AST check can tell a test that passes
models_used=[]explicitly from one that leaves the key out. Prevents: Finding 4 (test_no_recorded_model_renders_no_model_row at tests/test_experiment_reports.py:1404 says it covers 'an experiment.json written before models_used existed', but _make_result always sets models_used, so the missing-key default documented in docs/REPORT_SCHEMA.md never runs). - Add a fold-level model-identity test in tests/test_experiment_runner.py that drives _fold_replicates with realistic replicate results. Replicate 1 crashes before the first AssistantMessage, so the orchestrator.py:1136-1137 fallback records the configured alias 'claude-haiku-4-5'. Replicate 2 records the SDK-resolved 'claude-haiku-4-5-20251001'. Replicate 3 ends on a Claude Code '' message. Assert that VariantResult.models_used holds exactly one canonical id, and that experiment.md/html for a two-variant same-model experiment keep the Assistant Turns p-value and show no '(cross-model)' label. Why not static: The bad value appears only when runtime replicate states combine: crash-before-first-message fallback, SDK-resolved snapshot id, and synthetic final message. These come from the event stream and orchestrator control flow, not from a pattern in the source. Prevents: Finding 1 (raw '' and un-normalized ids saved by experiment.py:848 into experiment.json) and Finding 6 (dated snapshot vs alias read as two models, so a same-model A/B is marked cross-model and loses its turns p-value).
Top 5 Priority Actions
- Make the cross-model check use one canonical model identity. At src/coder_eval/reports/helpers.py:157,
normalize_modeldoes not map a dated snapshot to its alias, soturns_cross_model({'a': ['claude-haiku-4-5', 'claude-haiku-4-5-20251001'], 'b': ['claude-haiku-4-5-20251001']})returns True. As a result, a same-model experiment that has one crashed row (the config-alias fallback at src/coder_eval/orchestrator.py:1136-1137) loses the Welch turns p-value (reports/experiment.py:228 and html.py) and shows the 'more than one model' note. Map snapshots to their alias, or ignore the config fallback when another replicate has a resolved id, and add a test next totest_route_qualified_id_is_the_same_model. - Clean the model set at its source, not in the renderer. Stop Claude Code from recording
<synthetic>assdk_model_used(claude_code_agent.py:408-410). Normalize once in_fold_replicatesat src/coder_eval/orchestration/experiment.py:848, so the persisted experiment.jsonmodels_usedagrees with the report. Then delete_SYNTHETIC_MODELat src/coder_eval/reports/helpers.py:143 and makevariant_modelsa plain union. - Reword
CROSS_MODEL_TURNS_NOTEat src/coder_eval/reports/helpers.py:133 so it does not mention a p-value. The note is also shown for 1-variant and 3+-variant experiments, which have no p-value column (reports/experiment.py:252, reports/html.py:1375). As an alternative, show the 'no p-value' text only whenshow_p_valuesis true. - Add the missing negative HTML tests in tests/test_experiment_reports.py:1419. With
total_assistant_turns = Noneon every row, assert that the note and the 'Assistant Turns' row are absent (this tests theand has_turnsgate at src/coder_eval/reports/html.py:1459). Also assert that emptymodels_usedrenders no<td>Model</td>. In the same file at line 1404, maketest_no_recorded_model_renders_no_model_rowload a VariantResult that has nomodels_usedkey, or change its docstring. - Move the duplicated cross-model row logic into helpers, next to the shared note: the '(cross-model)' turns label (html.py:1443, reports/experiment.py:224) and the
', '.join(models) or 'n/a'model cell (html.py:1397, reports/experiment.py:273). Calculatehas_turnsone time in reports/experiment.py instead of twice (lines 223 and 293), so_experiment_aggregate_metrics(radon E) does not get more branches.
Stats: 0 🔴 · 0 🟠 · 2 🟡 · 4 🔵 across 8 axes reviewed.
…el check A row that replied records the resolved snapshot id while a row that crashed first keeps the configured alias, so a same-model A/B with one crashed row was labelled cross-model and lost its turns p-value. Drop the trailing snapshot date when comparing models. The cross-model note no longer mentions a p-value, since tables without a p-value column print it too. Adds tests for the snapshot case, the missing models_used key and the HTML no-turn-data and no-model cases.
Summary
VariantResultgainsmodels_used: the distinctmodel_usedvalues across the folded replicates, sorted. It defaults to[], so an olderexperiment.jsonstill loads.experiment.mdandexperiment.htmlgains a Model row. When the variants ran more than one model, the turns row reads Assistant Turns (cross-model), it has no p-value, and a note under the table says why. Score, duration and token rows keep their p-values: cost and wall clock stay comparable across models.eu.anthropic.claude-sonnet-5) is not a second model. Claude Code's<synthetic>placeholder (an error message the CLI makes up itself) is not a model either. A variant with no recorded model is unknown, never "different".pricing._normalize_modelis now public asnormalize_model, because the reports layer now uses it too. Callers and the two evalboard comments that name it are updated.Why
See #98. Turn counts depend on the model: one model chains a whole phase into one
Bashcall, another sends one probe per call. Before this change, a two-variant experiment on different models printed a turns row with a Welch p-value (illustrative numbers, from the issue's exhibit):A small p-value there reads as a real arm difference, but it is mostly model style. Now the same table reads:
On the same model the turns row and its p-value do not change, and the Model row shows the model.
Part of #98. This PR covers the experiment reports and the docs clause. The evalboard trend surfaces (per-task turn averages across runs) are not changed here. They need model grouping in the dashboard, which is a separate change.
Behaviour change
experiment.jsonrows carrymodels_used.experiment.md/experiment.htmlshow a Model row when any variant recorded a model. An experiment where no row recorded a model renders as before, so the existing report snapshots are unchanged.Docs
models_usedonVariantResult.Tests
tests/test_experiment_reports.py::TestCrossModelTurns(9 cases): same model keeps the p-value; different models mark the row, drop its p-value and add the note while duration keeps its p-value; a qualified route id is the same model;<synthetic>is not a model; mixed models inside one variant; an unrecorded model is unknown; no recorded model renders no Model row; no note without turn data; the HTML table marks the same rows as the markdown one.tests/test_experiment_runner.py: the replicate fold records each distinct model once, and[]when no replicate recorded one.coder-eval run tasks/agentless_smoke_test.yaml tasks/hello_date.yaml -e <exp>with twoagent.modelvariants. Withgpt-5.6-terravseu.anthropic.claude-sonnet-5,experiment.mdandexperiment.htmlshow the Model row, the cross-model turns row with—and the note. Withclaude-sonnet-5vseu.anthropic.claude-sonnet-5, the Model row showsclaude-sonnet-5twice and the turns row keeps its p-value. The errored claude-code row recordedmodel_used: <synthetic>, which is why that case is handled.tests/test_judge_litellm.py(10 tests) fails the same way onmainin my environment.