Skip to content

feat(criteria): add report_only to exclude a weighted criterion from scoring - #261

Open
NopeCharon wants to merge 1 commit into
UiPath:mainfrom
NopeCharon:feat/report-only-criteria
Open

NopeCharon wants to merge 1 commit into
UiPath:mainfrom
NopeCharon:feat/report-only-criteria

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

Summary

  • New report_only: true on every criterion. The criterion still runs, and its raw score, declared weight and pass_threshold stay in the record. It is left out of both final_status and weighted_score, and its result is persisted with gating: false.
  • BaseSuccessCriterion.is_gating is now weight > 0 and not report_only. This one property was already the source of truth for the gate, CriterionResult.gating, the evaluate summary, the HTML and JUnit counters, the suite-rollup failure reasons and the evalboard INFO pill, so those follow without their own changes.
  • calculate_weighted_score now skips non-gating criteria, so the score covers exactly the criteria the gate reads. A weight: 0 criterion already contributed nothing, so its results do not change.
  • Like weight: 0, report_only cannot be combined with a stop_early block or suite_thresholds. The validator now names the cause that applies (weight=0 or report_only: true).
  • Labels that said "weight: 0" (the HTML row note, the evaluate summary line) now just say informational, because a non-gating result no longer implies a zero weight.

Why

See #101. With the reproduction from the issue (A: score 1, weight 8, pass_threshold: 1; B: score 0, weight 1):

B is authored as final_status weighted_score B gating
pass_threshold: 0.0 (before and after) SUCCESS 0.889 (8/9) true
weight: 0 (before and after) SUCCESS 1.0 false, but the declared weight is lost
report_only: true (new) SUCCESS 1.0 false, weight stays 1

I chose an explicit field over giving pass_threshold: 0.0 a second meaning: a zero threshold already means "always passes", and changing that would silently re-score existing tasks.

Behaviour change

None for existing tasks: report_only defaults to false, and weight: 0 scoring is identical. Exported Harbor task.yaml files gain a report_only: false line per criterion (golden fixture regenerated), the same way weight: 1.0 is already written out.

Docs

  • Task Definition Guide: the field in the shared-fields table, an example, and how it differs from pass_threshold: 0.0. The weighted-score formula now says it runs over gating criteria.
  • Report Schema (gating), Extending, the task skill, and the lint-tasks skill (weight-at-risk totals leave out report_only criteria).
  • plugins/coder-eval/reference/criteria.md regenerated with make plugin-reference.

Tests

  • tests/test_threshold_enforcement.py: report-only excluded from score and gate at a non-zero weight (the issue's case), it cannot rescue a failing gating criterion, an all-report-only task, rejection with stop_early and with suite_thresholds, and the checker stamping gating: false.
  • tests/test_evaluate_command.py + a fixture task: evaluate end to end. Exit code 0, task.json has SUCCESS, weighted_score: 1.0 (not 8/9), and the report-only row has gating: false.
  • All 7 new or extended tests fail on main and pass here.
  • Locally (Ubuntu 24.04, Python 3.13): ruff format/check, pyright, custom lint (741 passed), prose budget, and pytest at 91% coverage. Outside this change: tests/test_judge_litellm.py (10 tests) fails the same way on main in my environment, and test_claude_agent_timeout_preserves_partial_turn_record failed once under -n 4 but passes when run alone.

Closes #101

…scoring

A criterion could only be made informational by setting weight: 0, which
rewrites the declared weight that some evaluation contracts must keep for
denominator or provenance parity. pass_threshold: 0.0 was no substitute:
it keeps the criterion out of final_status, but its score and weight
still count in weighted_score, and the result is persisted as gating.

Add report_only (default false) to every criterion. A report_only
criterion still runs and keeps its raw score, weight and pass_threshold
in the record, but BaseSuccessCriterion.is_gating is now false for it,
so the existing single source of truth carries it everywhere:

- calculate_weighted_score skips non-gating criteria, so the score covers
  exactly the criteria the gate reads (weight: 0 already contributed
  nothing, so its results are unchanged).
- all_criteria_passed, CriterionResult.gating, the evaluate summary, the
  HTML and JUnit reports, the suite rollup and the evalboard INFO pill
  all follow is_gating already.
- Like weight: 0, report_only may not be combined with a stop_early block
  or suite_thresholds; the validator now names whichever cause applies.

The surface labels that said "weight: 0" now say "informational", since
a non-gating result no longer implies a zero weight.

Docs: the Task Definition Guide documents the field with an example and
how it differs from pass_threshold: 0.0; the report schema, the
extending guide, the task and lint-tasks skills follow, and the plugin
criteria reference is regenerated. The Harbor export golden task.yaml
gains the new default field.

Closes UiPath#101

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coder_eval — pr:261

Scope: pr:261 · branch feat/report-only-criteria (PR #261 head; local main @ 4c88931a) · 076628b · 2026-11-14T13:00Z · workflow variant

Change class: complex — adds a public criterion schema field (report_only) that changes is_gating, the weighted score and the pass/fail gate semantics

coder_eval is healthy (9.8/10): type safety, security, architecture and error handling have no findings, and all confirmed findings come from one change (the new report_only flag), which adds a second way to mark a criterion as informational, makes a false "stored and rendered" claim about the weight it keeps, and gives a task with only informational criteria SUCCESS with weighted_score 0.0. Bottom line: remove report_only and keep weight: 0 as the only lever, then add tests for the empty-gate case and the CLI output.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 8.9 / 10 0 1 0 1 report_only duplicates weight: 0 as a second encoding of 'informational criterion'; the kept weight is never read
2. Type Safety 10 / 10 0 0 0 0 —
3. Test Health 9.8 / 10 0 0 0 2 The evaluate CLI test discards console output, so the changed informational summary and the ○ glyph are not asserted for report_only
4. Security 10 / 10 0 0 0 0 —
5. Architecture & Design 10 / 10 0 0 0 0 —
6. Error Handling & Resilience 10 / 10 0 0 0 0 —
7. API Surface & Maintainability 9.9 / 10 0 0 0 1 A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this
8. Evaluation Harness Quality 9.9 / 10 0 0 0 1 report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint

Overall Score: 9.8 / 10 · Weakest Axis: Code Quality & Style at 8.9 / 10
Totals: 🔴 0 · 🟠 1 · 🟡 0 · 🔵 5 across 8 axes.

Blockers

  1. [Axis 1] report_only duplicates weight: 0 as a second encoding of 'informational criterion'; the kept weight is never read (src/coder_eval/models/criteria.py:126) — The PR adds report_only: bool = Field(default=False, ...) (criteria.py:126) and changes is_gating to return self.weight > 0.0 and not self.report_only (criteria.py:231). weight: 0 already gives the same behaviour: the criterion is left out of the score and the gate, and its result is stored with gating: false. The only difference is that the declared weight stays in the YAML, and no code reads that weight when report_only is true. calculate_weighted_score skips it (if not criterion.is_gating: continue, results.py:768). all_criteria_passed filters it out (results.py:799). The validator stops it from being armed for early stop or suite thresholds. CriterionResult has no weight field. No renderer shows a criterion weight: grep of reports/ and evalboard/ for criterion weight finds nothing. The value only survives in the raw task_config snapshot. Because of this one boolean, "informational" now depends on two fields, the validator needs a two-way cause/remedy branch (criteria.py:204-207), and the weight docs, report_only docs, REPORT_SCHEMA, EXTENDING, the task skill, the lint-tasks skill and two evalboard comments all have to describe both levers and stay in sync. This is the rubric's 'user-facing knob that creates a second source of truth'. Preferred fix: delete report_only and keep weight: 0 as the only lever. Record the declared weight for provenance in the criterion description or a YAML comment. If a machine-readable declared weight is truly required for parity with another contract (#101), make it useful: persist it on CriterionResult and render it. Then there is one 'informational' predicate plus a value people can see, not a flag that keeps a number nothing reads.

Non-blocking, but please consider before merge

None.

Nits

  1. [Axis 1] Validator gives a wrong or incomplete remedy (and is untested) when weight: 0 and report_only: true are both set (src/coder_eval/models/criteria.py:204) — In criteria.py:204-207, if self.report_only: cause, remedy = "report_only: true", "Drop report_only" takes priority. For a criterion with weight: 0, report_only: true, stop_early: {...}, the error tells the author to drop report_only. If they do, weight: 0 still makes the criterion informational, and the next load fails again, this time blaming weight=0. Separately, weight: 0 with report_only: true is a redundant combination and is accepted without a warning (the inverse-guard check). If report_only is kept, either name both causes when both apply, or reject report_only: true together with weight: 0. Deleting report_only (see the first finding) removes this branch.
  2. [Axis 3] The evaluate CLI test discards console output, so the changed informational summary and the ○ glyph are not asserted for report_only (tests/test_evaluate_command.py:182) — The new test test_evaluate_command_report_only_criterion_is_excluded_from_gate_and_score patches the console with patch("coder_eval.cli.console.console.print"), (line 182). It asserts only the exit code and the task.json fields. The PR changed the CLI summary in evaluate_command.py: line 584 now prints console.print(f" [dim]Informational (not gated): {informational}[/dim]") and line 560 prints status = "[dim]○[/dim]" for non-gating rows. No test checks that a report_only row renders as ○ and not ✓/✗, or that the summary reads Passed: 1/1 with Informational (not gated): 1. The older weight-0 test at line 161 has the same gap. Fix: capture the mock's call args (or use a recording Console) and assert these three strings, so that a future edit that gates the display on weight == 0 instead of is_gating fails a test.
  3. [Axis 3] Five report_only tests repeat the five weight=0 tests; parametrize over the two informational forms (tests/test_threshold_enforcement.py:158) — Lines 158-212 add test_report_only_criterion_keeps_its_weight_but_neither_scores_nor_gates, test_report_only_does_not_rescue_a_failing_gating_criterion, test_all_report_only_criteria_score_zero_and_leave_an_empty_gate, test_report_only_cannot_be_armed_for_early_stop and test_report_only_cannot_declare_suite_thresholds. Each one copies the structure of its weight=0 twin at lines 109-156 and changes only the informational kwarg (weight=0.0 vs report_only=True) and the match string. Use one @pytest.mark.parametrize("informational_kwargs, cause", [({"weight": 0.0}, "weight=0"), ({"weight": 1.0, "report_only": True}, "report_only: true")]) over each scenario. This puts the 'both forms behave identically' contract in one place, and a third informational form later will need one new tuple instead of five new tests. Also, line 169 assert criteria[1].weight == 1.0 only reads back the constructor argument. If the 'keeps its declared weight' provenance contract matters, assert it on the persisted task_config.resolved in the evaluate-command test.
  4. [Axis 7] A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this (src/coder_eval/models/results.py:773) — Line 773: self.weighted_score = total_weighted_score / total_weight if total_weight > 0 else 0.0. The new if not criterion.is_gating: continue skips report_only criteria with non-zero weight, so a 'record everything, gate nothing' task now produces weighted_score 0.0. all_criteria_passed returns True for the empty gate, so the row is SUCCESS with score 0.0, and boards and trend lines read that as a total failure. TASK_DEFINITION_GUIDE.md now says the weighted score is 'over the gating criteria' but does not describe the empty case. Return None (the 'n/a' / unmeasured convention that format_score already supports) when no criterion is gating. If not, document the 0.0-on-SUCCESS result in the guide next to the report_only example. n/a
  5. [Axis 8] report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint (src/coder_eval/models/criteria.py:131) — The description reads "A report_only criterion still runs, and its raw score, weight and pass_threshold are " / "stored and rendered, ...". It is the SSOT, copied verbatim into plugins/coder-eval/reference/criteria.md. But CriterionResult (results.py:60-108) has no weight field, and no renderer outputs a criterion weight: grep weight in reports/html.py finds only font-weight CSS and weighted_score. The PR also changes html.py:590 from informational — not gated (weight: 0) to informational — not gated. A reader of task.json, the HTML report or the evalboard therefore cannot tell a report_only criterion from a weight-0 one, or see the declared weight that the field exists to preserve ("for provenance or parity"). The weight survives only in task_config.resolved. Either remove "and rendered" from the description (then run make plugin-reference), or persist the cause on the result, for example weight on CriterionResult, and render it in the advisory: informational — not gated (report_only, weight 1.0).

What's Missing

Parallel paths:

  • 🟡 The two guards that refuse to grade a task with no criteria were not updated for the new 'all criteria are informational' case. _reject_empty_criteria_under_grade (cli/run_command.py:990) and the regrade guard (orchestration/regrade.py:991) exist to stop a task that 'would silently score SUCCESS at weighted_score 0.0'. Both only check not task.success_criteria. A task whose criteria are all report_only: true (or all weight: 0) has an empty gate, passes both guards, and ends SUCCESS at 0.0, which is the result the guards were written to prevent. The PR documents report_only as a normal authoring pattern, so this case is now likely. Change both guards to test not any(c.is_gating for c in task.success_criteria). (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this)

Downstream consumers:

  • 🟡 Dialog mode reads the changed gate. orchestrator.py:2696 and :2769 pass all_criteria_passed(...) to simulation/termination.py evaluate_stop(criteria_all_passed=...). Take a task with stop_on_criteria_pass and per-turn checks whose criteria are all report_only. Its gate is empty and always true, so the dialog stops after turn 1 as if every criterion passed. No test covers this, and DIALOG_MODE.md does not mention it. (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this)
  • 🔵 harbor/reward.py:63 writes {"reward": result.weighted_score}. Because calculate_weighted_score now skips report_only criteria, an exported task whose criteria are all report_only sends Harbor reward 0.0 for a SUCCESS row. Harbor then reads it as a total failure, not as 'not scored'. (trigger: src/coder_eval/models/results.py) (restates: Axis 7: A task with only report_only criteria always ends SUCCESS with weighted_score 0.0, and the docs do not say this)

Tests:

  • 🔵 The PR changed the HTML advisory label from 'informational — not gated (weight: 0)' to 'informational — not gated' (reports/html.py:590). The header that reads 'N passed + M informational' (html.py:616-622) is untested for report_only rows. No test in tests/ contains 'not gated'. Add an HTML report test with one report_only result and one weight-0 result that checks the advisory text and the header counts. (trigger: src/coder_eval/reports/html.py)
  • 🔵 The Harbor export golden file only shows the default report_only: false. No test exports a task that has report_only: true and then grades it through the generated test.sh path (coder-eval evaluate followed by coder-eval harbor reward). So nothing confirms that the flag survives the export and that the reward leaves out the report_only criterion. _(trigger: tests/fixtures/harbor_export_golden/expected/tests/task.yaml)

Display & mapping dicts:

  • 🔵 The 'Criterion stats' table in suite.md (reports/markdown.py:969-984 and 1114-1122) averages scores by criterion_type and has no gating column. CriterionStats carries no gating flag. A report_only criterion's misses therefore go into the same row as a gating criterion of the same type, and nothing marks the row as informational. weight-0 criteria already had this problem. report_only makes it more common because it invites keeping a non-zero weight on side checks. (trigger: src/coder_eval/reports/markdown.py)
  • 🔵 No rendering surface tells report_only apart from weight 0, and none shows the declared weight: the evalboard INFO pill (_chips.tsx), the HTML advisory (html.py:590, where the PR removed the '(weight: 0)' cause), the CLI ○ row, and JUnit. CriterionResult has no weight field and no cause field. So the declared weight that report_only exists to keep cannot be seen on any surface. _(trigger: evalboard/app/runs/[id]/[...task]/chips.tsx) (restates: Axis 8: report_only description says the weight is 'stored and rendered', but no surface renders it, and the HTML report loses its cause hint)

Daily/nightly:

  • 🔵 The PR does not say what happens to the nightly or downstream pipeline. For existing tasks, weighted_score does not change, because no task in the corpus sets report_only. But every criterion is now written with report_only: false in two places: each task.json task_config.resolved (orchestrator.py:1171, model_dump with defaults) and each Harbor export (harbor/packager.py:367, which uses exclude_none only). Criteria use extra="forbid", so an older coder-eval will reject these records. That covers a detached evaluate <run_dir> and the Harbor verifier's coder-eval evaluate /tests/task.yaml when either runs an older version. Fix: state the minimum reader version in the PR or CHANGELOG, or leave the field out of the dump when it has its default value. (trigger: src/coder_eval/models/criteria.py)

Harness & Lint Improvements

Static checks (lint / type):

  • [ce-lint] New rule CE069 ce069_gating_via_is_gating.py in tests/lint/rules/, wired in tests/lint/runner.py. It forbids a Compare node where one side is an attribute named weight and the other side is the numeric literal 0 / 0.0 (operators ==, !=, >, >=, <, <=). The rule applies everywhere in src/coder_eval/ except BaseSuccessCriterion in models/criteria.py. Only criterion.is_gating may decide whether a criterion is informational. The comparison must never be repeated as weight == 0 / weight > 0 in a renderer, CLI summary, aggregator or evalboard-feeding serializer. Local totals such as total_weight > 0 in results.py are bare names, not .weight attributes, so they do not fire. On main, the only hits are criteria.py:189 and :213, and both are inside the exempt class. Prevents: Finding 3 (evaluate CLI): a later edit that gates the ○ glyph or the 'Informational (not gated)' count on weight == 0 and not on is_gating would quietly show report_only rows as gated ✓/✗, and the test cannot catch it because it discards console output. It also limits finding 1's 'second encoding' problem: while two levers exist, every reader goes through the single predicate. The rule does NOT find the duplicate knob itself. Whether two fields encode one concept is a semantic judgment.
  • [ce-lint] Widen CE049 (ce049_no_score_or_zero.py) from <score> or <numeric literal> to the conditional form as well: an IfExp whose orelse (or body) is a numeric Constant, where the expression is assigned to or returned into a target whose trailing name matches the existing _SCORE_NAME regex (weighted_score, score, average_score, pass_rate, *_rate). Example: self.weighted_score = total / total_weight if total_weight > 0 else 0.0. The fix is None (unmeasured) or an explicit # noqa: CE049 that names why an empty denominator is a real miss. Audit the current hits before you enable the rule. The cap-rule convention applies: fix or noqa every hit in the same change. Prevents: Finding 5 (results.py:773): with no gating criterion, a task with only report_only criteria (or only weight-0 criteria) writes weighted_score = 0.0 on a SUCCESS row, and boards read that as a total failure. CE049 already encodes the 'None means unmeasured, 0.0 means measured-zero' contract, but it misses the ternary spelling of the same coalesce.

Harness improvements (not statically reachable):

  • Add a combinatorial validator contract test in tests/test_threshold_enforcement.py. Build the cross product of the informational levers ({weight: 0}, {report_only: true}, both) and the features that are not allowed on an informational criterion (stop_early, suite_thresholds). Assert two things: (a) every combination is rejected, and (b) the error message names EVERY active cause, so that following the stated remedy makes the YAML load. Also assert that the redundant weight: 0 + report_only: true pair is rejected or warned about. If finding 1's preferred fix (delete report_only) lands, the matrix becomes one row and this test becomes moot. Why not static: Whether a remedy string is complete depends on what the validator does at runtime with each field combination. The text of a remedy cannot be checked against the branches it should name with an AST check. Prevents: Finding 2 (criteria.py:204-207): the 'Drop report_only' remedy is wrong when weight: 0 is also set, and that combination has no test.
  • Add a shared recording_console pytest fixture in tests/conftest.py. It swaps coder_eval.cli.console.console for a rich.console.Console(record=True, width=200) and returns the exported text. Use it in the evaluate-command tests for both informational forms. Assert the ○ glyph on the informational row, Passed: 1/1, and Informational (not gated): 1. Keep the bare patch(...console.print) mute for tests that really do not care about output. Why not static: Muting the console is legitimate in about 12 existing test sites, so whether a given test MUST assert rendered output is a judgment about what that test is for. A lint rule on the bare patch would mostly report false positives. Prevents: Finding 3 (tests/test_evaluate_command.py:182 and the older weight-0 test at :161): the changed CLI summary and the ○ glyph have no test.
  • Add one module-level INFORMATIONAL_FORMS parametrize table (kwargs, expected cause string) to tests/test_threshold_enforcement.py. Every 'informational criterion' scenario (scores nothing, does not rescue a failing gate, all-informational gives an empty gate, cannot be armed, cannot declare suite_thresholds) runs over that table and is not copied per lever. Record this in the test-writing guidance: 'N equivalent encodings of one behaviour means one parametrize table, not N copies'. Why not static: To find near-duplicate test bodies you need AST-similarity heuristics with a tuned threshold. Such a check is noisy and the current lint toolchain does not have one. Whether two tests MEAN to assert the same contract is semantic. Prevents: Finding 4 (tests/test_threshold_enforcement.py:158-212): five report_only tests copy their weight=0 twins.
  • Add a render-provenance golden test: a fixture task with one gating criterion, one weight-0 criterion and one report_only criterion (weight 1.0) goes through task.json, run.md and reports/html.py. The test asserts exactly what the criterion Field descriptions promise is visible: the 'informational — not gated' advisory, its cause, and the declared weight if the description says it is rendered. Pair it with a parity check: each BaseSuccessCriterion Field whose description says 'stored' or 'rendered' must have a matching field on CriterionResult or a renderer reference, or an EXEMPT entry with a reason (the same shape as CE030's EXEMPT table). Also promote the existing harness-candidates item 'Extend CE030 doc/schema-parity to the SuccessCriterion union', so that a new criterion-level knob is forced through a doc and design review. Why not static: Whether a field's description is TRUE about what renders needs the rendered output of a real result. A keyword-to-renderer grep would be fragile. The core of finding 1 ('two fields encode one concept; the kept weight is never read') is a design judgment that no checker can make. Prevents: Finding 6 (criteria.py:129-131): the false 'weight ... stored and rendered' claim, copied into plugins/coder-eval/reference/criteria.md. Also finding 1: a preserved weight that no surface reads would have failed the golden test.
  • Add an empty-gate scoring-determinism test: a task where no criterion gates (all weight 0, or all report_only) must produce the documented weighted_score (preferably None, rendered as 'n/a' by format_score). The test also asserts the final status and how the row enters suite pass rates and run.md's averages. Why not static: The defect comes from evaluating the scoring path on a particular criteria mix (an empty gating set) and from how downstream aggregators treat that value. The widened CE049 catches the coalesce spelling, but not the end-to-end SUCCESS-with-0.0 meaning. Prevents: Finding 5 (results.py:773, and the test at tests/test_threshold_enforcement.py:183 that locks in the 0.0 result).

Top 5 Priority Actions

  1. Fix the empty-gate score at src/coder_eval/models/results.py:773: when no criterion is gating, return None (shown as n/a by format_score) and not 0.0, so that a task with only informational criteria does not show as SUCCESS with score 0.0 on boards and trend lines; if you keep 0.0, document it in TASK_DEFINITION_GUIDE.md next to the report_only example and update tests/test_threshold_enforcement.py:183.
  2. Delete report_only at src/coder_eval/models/criteria.py:126 and the and not self.report_only term in is_gating (criteria.py:231), so that weight: 0 is the only way to mark a criterion as informational; also remove the matching text from the guide, REPORT_SCHEMA, EXTENDING, the task and lint-tasks skills and the evalboard comments, and run make plugin-reference.
  3. If you keep report_only, correct the false claim in its description at src/coder_eval/models/criteria.py:131 ("weight and pass_threshold are stored and rendered"): either remove "and rendered", or add a weight field to CriterionResult and show the cause in the HTML advisory at src/coder_eval/reports/html.py:590 (for example "report_only, weight 1.0"); then run make plugin-reference.
  4. If you keep report_only, fix the validator at src/coder_eval/models/criteria.py:204-207 so that it names both causes when weight: 0 and report_only: true are both set (or rejects that combination), and add a test for it, because the current "Drop report_only" remedy leads to a second failure on the next load.
  5. Make the tests stronger: in tests/test_evaluate_command.py:182 (and the weight-0 test at line 161), record the console output and assert the ○ glyph, Passed: 1/1 and Informational (not gated): 1; in tests/test_threshold_enforcement.py:158-212, combine the five report_only tests and their five weight=0 twins into tests parametrized over the two informational forms.

Stats: 0 🔴 · 1 🟠 · 0 🟡 · 5 🔵 across 8 axes reviewed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support report-only criteria without discarding their declared weight

2 participants