Repository navigation
feat(criteria): add report_only to exclude a weighted criterion from scoring - #261
NopeCharon wants to merge 1 commit into
Conversation
…scoring A criterion could only be made informational by setting weight: 0, which rewrites the declared weight that some evaluation contracts must keep for denominator or provenance parity. pass_threshold: 0.0 was no substitute: it keeps the criterion out of final_status, but its score and weight still count in weighted_score, and the result is persisted as gating. Add report_only (default false) to every criterion. A report_only criterion still runs and keeps its raw score, weight and pass_threshold in the record, but BaseSuccessCriterion.is_gating is now false for it, so the existing single source of truth carries it everywhere: - calculate_weighted_score skips non-gating criteria, so the score covers exactly the criteria the gate reads (weight: 0 already contributed nothing, so its results are unchanged). - all_criteria_passed, CriterionResult.gating, the evaluate summary, the HTML and JUnit reports, the suite rollup and the evalboard INFO pill all follow is_gating already. - Like weight: 0, report_only may not be combined with a stop_early block or suite_thresholds; the validator now names whichever cause applies. The surface labels that said "weight: 0" now say "informational", since a non-gating result no longer implies a zero weight. Docs: the Task Definition Guide documents the field with an example and how it differs from pass_threshold: 0.0; the report schema, the extending guide, the task and lint-tasks skills follow, and the plugin criteria reference is regenerated. The Harbor export golden task.yaml gains the new default field. Closes UiPath#101
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:261
Scope: pr:261 · branch feat/report-only-criteria (PR #261 head; local main @ 4c88931a) · 076628b · 2026-11-14T13:00Z · workflow variant
Change class: complex — adds a public criterion schema field (report_only) that changes is_gating, the weighted score and the pass/fail gate semantics
coder_eval is healthy (9.8/10): type safety, security, architecture and error handling have no findings, and all confirmed findings come from one change (the new report_only flag), which adds a second way to mark a criterion as informational, makes a false "stored and rendered" claim about the weight it keeps, and gives a task with only informational criteria SUCCESS with weighted_score 0.0. Bottom line: remove report_only and keep weight: 0 as the only lever, then add tests for the empty-gate case and the CLI output.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 8.9 / 10 | 0 | 1 | 0 | 1 | report_only duplicates weight: 0 as a second encoding of 'informational criterion'; the kept weight is never read |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 9.8 / 10 | 0 | 0 | 0 | 2 | The evaluate CLI test discards console output, so the changed informational summary and the ○ glyph are not asserted for report_only |
| 4. Security | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 7. API Surface & Maintainability | 9.9 / 10 | 0 | 0 | 0 | 1 | A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this |
| 8. Evaluation Harness Quality | 9.9 / 10 | 0 | 0 | 0 | 1 | report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint |
Overall Score: 9.8 / 10 · Weakest Axis: Code Quality & Style at 8.9 / 10
Totals: 🔴 0 · 🟠 1 · 🟡 0 · 🔵 5 across 8 axes.
Blockers
- [Axis 1]
report_onlyduplicatesweight: 0as a second encoding of 'informational criterion'; the kept weight is never read (src/coder_eval/models/criteria.py:126) — The PR addsreport_only: bool = Field(default=False, ...)(criteria.py:126) and changesis_gatingtoreturn self.weight > 0.0 and not self.report_only(criteria.py:231).weight: 0already gives the same behaviour: the criterion is left out of the score and the gate, and its result is stored withgating: false. The only difference is that the declared weight stays in the YAML, and no code reads that weight whenreport_onlyis true.calculate_weighted_scoreskips it (if not criterion.is_gating: continue, results.py:768).all_criteria_passedfilters it out (results.py:799). The validator stops it from being armed for early stop or suite thresholds.CriterionResulthas noweightfield. No renderer shows a criterion weight: grep of reports/ and evalboard/ for criterionweightfinds nothing. The value only survives in the rawtask_configsnapshot. Because of this one boolean, "informational" now depends on two fields, the validator needs a two-way cause/remedy branch (criteria.py:204-207), and the weight docs,report_onlydocs, REPORT_SCHEMA, EXTENDING, the task skill, the lint-tasks skill and two evalboard comments all have to describe both levers and stay in sync. This is the rubric's 'user-facing knob that creates a second source of truth'. Preferred fix: deletereport_onlyand keepweight: 0as the only lever. Record the declared weight for provenance in the criteriondescriptionor a YAML comment. If a machine-readable declared weight is truly required for parity with another contract (#101), make it useful: persist it onCriterionResultand render it. Then there is one 'informational' predicate plus a value people can see, not a flag that keeps a number nothing reads.
Non-blocking, but please consider before merge
None.
Nits
- [Axis 1] Validator gives a wrong or incomplete remedy (and is untested) when weight: 0 and report_only: true are both set (
src/coder_eval/models/criteria.py:204) — In criteria.py:204-207,if self.report_only: cause, remedy = "report_only: true", "Drop report_only"takes priority. For a criterion withweight: 0, report_only: true, stop_early: {...}, the error tells the author to dropreport_only. If they do,weight: 0still makes the criterion informational, and the next load fails again, this time blamingweight=0. Separately,weight: 0withreport_only: trueis a redundant combination and is accepted without a warning (the inverse-guard check). Ifreport_onlyis kept, either name both causes when both apply, or rejectreport_only: truetogether withweight: 0. Deletingreport_only(see the first finding) removes this branch. - [Axis 3] The evaluate CLI test discards console output, so the changed informational summary and the ○ glyph are not asserted for report_only (
tests/test_evaluate_command.py:182) — The new testtest_evaluate_command_report_only_criterion_is_excluded_from_gate_and_scorepatches the console withpatch("coder_eval.cli.console.console.print"),(line 182). It asserts only the exit code and the task.json fields. The PR changed the CLI summary in evaluate_command.py: line 584 now printsconsole.print(f" [dim]Informational (not gated): {informational}[/dim]")and line 560 printsstatus = "[dim]○[/dim]"for non-gating rows. No test checks that a report_only row renders as ○ and not ✓/✗, or that the summary readsPassed: 1/1withInformational (not gated): 1. The older weight-0 test at line 161 has the same gap. Fix: capture the mock's call args (or use a recording Console) and assert these three strings, so that a future edit that gates the display onweight == 0instead ofis_gatingfails a test. - [Axis 3] Five report_only tests repeat the five weight=0 tests; parametrize over the two informational forms (
tests/test_threshold_enforcement.py:158) — Lines 158-212 addtest_report_only_criterion_keeps_its_weight_but_neither_scores_nor_gates,test_report_only_does_not_rescue_a_failing_gating_criterion,test_all_report_only_criteria_score_zero_and_leave_an_empty_gate,test_report_only_cannot_be_armed_for_early_stopandtest_report_only_cannot_declare_suite_thresholds. Each one copies the structure of its weight=0 twin at lines 109-156 and changes only the informational kwarg (weight=0.0vsreport_only=True) and the match string. Use one@pytest.mark.parametrize("informational_kwargs, cause", [({"weight": 0.0}, "weight=0"), ({"weight": 1.0, "report_only": True}, "report_only: true")])over each scenario. This puts the 'both forms behave identically' contract in one place, and a third informational form later will need one new tuple instead of five new tests. Also, line 169assert criteria[1].weight == 1.0only reads back the constructor argument. If the 'keeps its declared weight' provenance contract matters, assert it on the persistedtask_config.resolvedin the evaluate-command test. - [Axis 7] A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this (
src/coder_eval/models/results.py:773) — Line 773:self.weighted_score = total_weighted_score / total_weight if total_weight > 0 else 0.0. The newif not criterion.is_gating: continueskips report_only criteria with non-zero weight, so a 'record everything, gate nothing' task now produces weighted_score 0.0.all_criteria_passedreturns True for the empty gate, so the row is SUCCESS with score 0.0, and boards and trend lines read that as a total failure. TASK_DEFINITION_GUIDE.md now says the weighted score is 'over the gating criteria' but does not describe the empty case. ReturnNone(the 'n/a' / unmeasured convention thatformat_scorealready supports) when no criterion is gating. If not, document the 0.0-on-SUCCESS result in the guide next to the report_only example. n/a - [Axis 8] report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint (
src/coder_eval/models/criteria.py:131) — The description reads "A report_only criterion still runs, and its raw score, weight and pass_threshold are " / "stored and rendered, ...". It is the SSOT, copied verbatim into plugins/coder-eval/reference/criteria.md. ButCriterionResult(results.py:60-108) has noweightfield, and no renderer outputs a criterion weight: grepweightin reports/html.py finds onlyfont-weightCSS andweighted_score. The PR also changes html.py:590 frominformational — not gated (weight: 0)toinformational — not gated. A reader of task.json, the HTML report or the evalboard therefore cannot tell a report_only criterion from a weight-0 one, or see the declared weight that the field exists to preserve ("for provenance or parity"). The weight survives only intask_config.resolved. Either remove "and rendered" from the description (then runmake plugin-reference), or persist the cause on the result, for exampleweighton CriterionResult, and render it in the advisory:informational — not gated (report_only, weight 1.0).
What's Missing
Parallel paths:
- 🟡 The two guards that refuse to grade a task with no criteria were not updated for the new 'all criteria are informational' case.
_reject_empty_criteria_under_grade(cli/run_command.py:990) and the regrade guard (orchestration/regrade.py:991) exist to stop a task that 'would silently score SUCCESS at weighted_score 0.0'. Both only checknot task.success_criteria. A task whose criteria are allreport_only: true(or allweight: 0) has an empty gate, passes both guards, and ends SUCCESS at 0.0, which is the result the guards were written to prevent. The PR documents report_only as a normal authoring pattern, so this case is now likely. Change both guards to testnot any(c.is_gating for c in task.success_criteria). (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this)
Downstream consumers:
- 🟡 Dialog mode reads the changed gate. orchestrator.py:2696 and :2769 pass
all_criteria_passed(...)to simulation/termination.pyevaluate_stop(criteria_all_passed=...). Take a task withstop_on_criteria_passand per-turn checks whose criteria are all report_only. Its gate is empty and always true, so the dialog stops after turn 1 as if every criterion passed. No test covers this, and DIALOG_MODE.md does not mention it. (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this) - 🔵 harbor/reward.py:63 writes
{"reward": result.weighted_score}. Becausecalculate_weighted_scorenow skips report_only criteria, an exported task whose criteria are all report_only sends Harbor reward 0.0 for a SUCCESS row. Harbor then reads it as a total failure, not as 'not scored'. (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this)
Tests:
- 🔵 The PR changed the HTML advisory label from 'informational — not gated (weight: 0)' to 'informational — not gated' (reports/html.py:590). The header that reads 'N passed + M informational' (html.py:616-622) is untested for report_only rows. No test in tests/ contains 'not gated'. Add an HTML report test with one report_only result and one weight-0 result that checks the advisory text and the header counts. (trigger: src/coder_eval/reports/html.py)
- 🔵 The Harbor export golden file only shows the default
report_only: false. No test exports a task that hasreport_only: trueand then grades it through the generated test.sh path (coder-eval evaluatefollowed bycoder-eval harbor reward). So nothing confirms that the flag survives the export and that the reward leaves out the report_only criterion. _(trigger: tests/fixtures/harbor_export_golden/expected/tests/task.yaml)
Display & mapping dicts:
- 🔵 The 'Criterion stats' table in suite.md (reports/markdown.py:969-984 and 1114-1122) averages scores by
criterion_typeand has no gating column.CriterionStatscarries no gating flag. A report_only criterion's misses therefore go into the same row as a gating criterion of the same type, and nothing marks the row as informational. weight-0 criteria already had this problem. report_only makes it more common because it invites keeping a non-zero weight on side checks. (trigger: src/coder_eval/reports/markdown.py) - 🔵 No rendering surface tells report_only apart from weight 0, and none shows the declared weight: the evalboard INFO pill (_chips.tsx), the HTML advisory (html.py:590, where the PR removed the '(weight: 0)' cause), the CLI ○ row, and JUnit.
CriterionResulthas no weight field and no cause field. So the declared weight that report_only exists to keep cannot be seen on any surface. _(trigger: evalboard/app/runs/[id]/[...task]/chips.tsx) (restates: Axis 8: report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint)
Daily/nightly:
- 🔵 The PR does not say what happens to the nightly or downstream pipeline. For existing tasks, weighted_score does not change, because no task in the corpus sets report_only. But every criterion is now written with
report_only: falsein two places: each task.jsontask_config.resolved(orchestrator.py:1171,model_dumpwith defaults) and each Harbor export (harbor/packager.py:367, which usesexclude_noneonly). Criteria useextra="forbid", so an older coder-eval will reject these records. That covers a detachedevaluate <run_dir>and the Harbor verifier'scoder-eval evaluate /tests/task.yamlwhen either runs an older version. Fix: state the minimum reader version in the PR or CHANGELOG, or leave the field out of the dump when it has its default value. (trigger: src/coder_eval/models/criteria.py)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] New rule CE069
ce069_gating_via_is_gating.pyin tests/lint/rules/, wired in tests/lint/runner.py. It forbids aComparenode where one side is an attribute namedweightand the other side is the numeric literal 0 / 0.0 (operators ==, !=, >, >=, <, <=). The rule applies everywhere in src/coder_eval/ exceptBaseSuccessCriterionin models/criteria.py. Onlycriterion.is_gatingmay decide whether a criterion is informational. The comparison must never be repeated asweight == 0/weight > 0in a renderer, CLI summary, aggregator or evalboard-feeding serializer. Local totals such astotal_weight > 0in results.py are bare names, not.weightattributes, so they do not fire. On main, the only hits are criteria.py:189 and :213, and both are inside the exempt class. Prevents: Finding 3 (evaluate CLI): a later edit that gates the ○ glyph or the 'Informational (not gated)' count onweight == 0and not onis_gatingwould quietly show report_only rows as gated ✓/✗, and the test cannot catch it because it discards console output. It also limits finding 1's 'second encoding' problem: while two levers exist, every reader goes through the single predicate. The rule does NOT find the duplicate knob itself. Whether two fields encode one concept is a semantic judgment. - [ce-lint] Widen CE049 (
ce049_no_score_or_zero.py) from<score> or <numeric literal>to the conditional form as well: anIfExpwhoseorelse(orbody) is a numericConstant, where the expression is assigned to or returned into a target whose trailing name matches the existing_SCORE_NAMEregex (weighted_score, score, average_score, pass_rate, *_rate). Example:self.weighted_score = total / total_weight if total_weight > 0 else 0.0. The fix isNone(unmeasured) or an explicit# noqa: CE049that names why an empty denominator is a real miss. Audit the current hits before you enable the rule. The cap-rule convention applies: fix or noqa every hit in the same change. Prevents: Finding 5 (results.py:773): with no gating criterion, a task with only report_only criteria (or only weight-0 criteria) writesweighted_score = 0.0on a SUCCESS row, and boards read that as a total failure. CE049 already encodes the 'None means unmeasured, 0.0 means measured-zero' contract, but it misses the ternary spelling of the same coalesce.
Harness improvements (not statically reachable):
- Add a combinatorial validator contract test in tests/test_threshold_enforcement.py. Build the cross product of the informational levers ({weight: 0}, {report_only: true}, both) and the features that are not allowed on an informational criterion (stop_early, suite_thresholds). Assert two things: (a) every combination is rejected, and (b) the error message names EVERY active cause, so that following the stated remedy makes the YAML load. Also assert that the redundant
weight: 0+report_only: truepair is rejected or warned about. If finding 1's preferred fix (delete report_only) lands, the matrix becomes one row and this test becomes moot. Why not static: Whether a remedy string is complete depends on what the validator does at runtime with each field combination. The text of a remedy cannot be checked against the branches it should name with an AST check. Prevents: Finding 2 (criteria.py:204-207): the 'Drop report_only' remedy is wrong when weight: 0 is also set, and that combination has no test. - Add a shared
recording_consolepytest fixture in tests/conftest.py. It swapscoder_eval.cli.console.consolefor arich.console.Console(record=True, width=200)and returns the exported text. Use it in the evaluate-command tests for both informational forms. Assert the ○ glyph on the informational row,Passed: 1/1, andInformational (not gated): 1. Keep the barepatch(...console.print)mute for tests that really do not care about output. Why not static: Muting the console is legitimate in about 12 existing test sites, so whether a given test MUST assert rendered output is a judgment about what that test is for. A lint rule on the bare patch would mostly report false positives. Prevents: Finding 3 (tests/test_evaluate_command.py:182 and the older weight-0 test at :161): the changed CLI summary and the ○ glyph have no test. - Add one module-level
INFORMATIONAL_FORMSparametrize table (kwargs, expected cause string) to tests/test_threshold_enforcement.py. Every 'informational criterion' scenario (scores nothing, does not rescue a failing gate, all-informational gives an empty gate, cannot be armed, cannot declare suite_thresholds) runs over that table and is not copied per lever. Record this in the test-writing guidance: 'N equivalent encodings of one behaviour means one parametrize table, not N copies'. Why not static: To find near-duplicate test bodies you need AST-similarity heuristics with a tuned threshold. Such a check is noisy and the current lint toolchain does not have one. Whether two tests MEAN to assert the same contract is semantic. Prevents: Finding 4 (tests/test_threshold_enforcement.py:158-212): five report_only tests copy their weight=0 twins. - Add a render-provenance golden test: a fixture task with one gating criterion, one weight-0 criterion and one report_only criterion (weight 1.0) goes through task.json, run.md and reports/html.py. The test asserts exactly what the criterion Field descriptions promise is visible: the 'informational — not gated' advisory, its cause, and the declared weight if the description says it is rendered. Pair it with a parity check: each
BaseSuccessCriterionField whose description says 'stored' or 'rendered' must have a matching field onCriterionResultor a renderer reference, or an EXEMPT entry with a reason (the same shape as CE030's EXEMPT table). Also promote the existing harness-candidates item 'Extend CE030 doc/schema-parity to the SuccessCriterion union', so that a new criterion-level knob is forced through a doc and design review. Why not static: Whether a field's description is TRUE about what renders needs the rendered output of a real result. A keyword-to-renderer grep would be fragile. The core of finding 1 ('two fields encode one concept; the kept weight is never read') is a design judgment that no checker can make. Prevents: Finding 6 (criteria.py:129-131): the false 'weight ... stored and rendered' claim, copied into plugins/coder-eval/reference/criteria.md. Also finding 1: a preserved weight that no surface reads would have failed the golden test. - Add an empty-gate scoring-determinism test: a task where no criterion gates (all weight 0, or all report_only) must produce the documented
weighted_score(preferably None, rendered as 'n/a' byformat_score). The test also asserts the final status and how the row enters suite pass rates and run.md's averages. Why not static: The defect comes from evaluating the scoring path on a particular criteria mix (an empty gating set) and from how downstream aggregators treat that value. The widened CE049 catches the coalesce spelling, but not the end-to-end SUCCESS-with-0.0 meaning. Prevents: Finding 5 (results.py:773, and the test at tests/test_threshold_enforcement.py:183 that locks in the 0.0 result).
Top 5 Priority Actions
- Fix the empty-gate score at src/coder_eval/models/results.py:773: when no criterion is gating, return
None(shown as n/a byformat_score) and not 0.0, so that a task with only informational criteria does not show as SUCCESS with score 0.0 on boards and trend lines; if you keep 0.0, document it in TASK_DEFINITION_GUIDE.md next to the report_only example and update tests/test_threshold_enforcement.py:183. - Delete
report_onlyat src/coder_eval/models/criteria.py:126 and theand not self.report_onlyterm inis_gating(criteria.py:231), so thatweight: 0is the only way to mark a criterion as informational; also remove the matching text from the guide, REPORT_SCHEMA, EXTENDING, the task and lint-tasks skills and the evalboard comments, and runmake plugin-reference. - If you keep
report_only, correct the false claim in its description at src/coder_eval/models/criteria.py:131 ("weight and pass_threshold are stored and rendered"): either remove "and rendered", or add aweightfield toCriterionResultand show the cause in the HTML advisory at src/coder_eval/reports/html.py:590 (for example "report_only, weight 1.0"); then runmake plugin-reference. - If you keep
report_only, fix the validator at src/coder_eval/models/criteria.py:204-207 so that it names both causes whenweight: 0andreport_only: trueare both set (or rejects that combination), and add a test for it, because the current "Drop report_only" remedy leads to a second failure on the next load. - Make the tests stronger: in tests/test_evaluate_command.py:182 (and the weight-0 test at line 161), record the console output and assert the ○ glyph,
Passed: 1/1andInformational (not gated): 1; in tests/test_threshold_enforcement.py:158-212, combine the five report_only tests and their five weight=0 twins into tests parametrized over the two informational forms.
Stats: 0 🔴 · 1 🟠 · 0 🟡 · 5 🔵 across 8 axes reviewed.
Summary
report_only: trueon every criterion. The criterion still runs, and its raw score, declaredweightandpass_thresholdstay in the record. It is left out of bothfinal_statusandweighted_score, and its result is persisted withgating: false.BaseSuccessCriterion.is_gatingis nowweight > 0 and not report_only. This one property was already the source of truth for the gate,CriterionResult.gating, theevaluatesummary, the HTML and JUnit counters, the suite-rollup failure reasons and the evalboardINFOpill, so those follow without their own changes.calculate_weighted_scorenow skips non-gating criteria, so the score covers exactly the criteria the gate reads. Aweight: 0criterion already contributed nothing, so its results do not change.weight: 0,report_onlycannot be combined with astop_earlyblock orsuite_thresholds. The validator now names the cause that applies (weight=0orreport_only: true).evaluatesummary line) now just say informational, because a non-gating result no longer implies a zero weight.Why
See #101. With the reproduction from the issue (A: score 1, weight 8,
pass_threshold: 1; B: score 0, weight 1):final_statusweighted_scoregatingpass_threshold: 0.0(before and after)trueweight: 0(before and after)false, but the declared weight is lostreport_only: true(new)false, weight stays 1I chose an explicit field over giving
pass_threshold: 0.0a second meaning: a zero threshold already means "always passes", and changing that would silently re-score existing tasks.Behaviour change
None for existing tasks:
report_onlydefaults tofalse, andweight: 0scoring is identical. Exported Harbortask.yamlfiles gain areport_only: falseline per criterion (golden fixture regenerated), the same wayweight: 1.0is already written out.Docs
pass_threshold: 0.0. The weighted-score formula now says it runs over gating criteria.gating), Extending, thetaskskill, and thelint-tasksskill (weight-at-risk totals leave outreport_onlycriteria).plugins/coder-eval/reference/criteria.mdregenerated withmake plugin-reference.Tests
tests/test_threshold_enforcement.py: report-only excluded from score and gate at a non-zero weight (the issue's case), it cannot rescue a failing gating criterion, an all-report-only task, rejection withstop_earlyand withsuite_thresholds, and the checker stampinggating: false.tests/test_evaluate_command.py+ a fixture task:evaluateend to end. Exit code 0,task.jsonhasSUCCESS,weighted_score: 1.0(not 8/9), and the report-only row hasgating: false.mainand pass here.tests/test_judge_litellm.py(10 tests) fails the same way onmainin my environment, andtest_claude_agent_timeout_preserves_partial_turn_recordfailed once under-n 4but passes when run alone.Closes #101