Repository navigation
fix(skillsbench): reject incomplete court forms - #260
Conversation
…g-fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pypdf 6.21.4 has five GHSA advisories fixed only in 7.0.0/7.0.1 (GHSA-cmpx-464x-82c3 XFA memory use, GHSA-rfv5-695j-j993, GHSA-fhff-j4m8-439h, GHSA-59jv-x5x2-gg4c, GHSA-mwj3-fpmc-hjcv). Bump the dev and Dockerfile pins to 7.0.1, and the image's cryptography pin from 43.0.3 to 50.0.2. - Add extract_field_values to the verifier so the tests call plain functions instead of the pytest-internal fixture __wrapped__. - Add a test that keeps the Dockerfile and dev pypdf pins equal. - README: state that PyPDF2 3.0.1 stays and has unfixed advisories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @uipreliga's task in 1m 13s —— View job Code Review in Progress
|
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:260
Scope: pr:260 · branch fix/court-form-missing-field (PR #260 head) · 4c62954 · 2026-11-13T23:10Z · workflow variant
Change class: complex — changes the court-form verifier's pass/fail control flow (skip paths removed, missing fields now fail, extraction errors propagate), which changes task reward for identical agent output
The codebase is healthy (9.6/10): there are no critical or high findings, and the reward gate cannot give a false pass because test.sh writes reward 1 only when pytest exits 0, but the patched court-form verifier can still give a 0 to correct agent output, for two reasons: a broken XFA parser whose error it hides, and a stricter grading contract that the docs do not state; the bottom line is to fix those two first, then close the CE071 coverage gaps.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 8.7 / 10 | 0 | 0 | 2 | 3 | CE071 _offenders is one 3-clause boolean comprehension with cyclomatic complexity 17 (radon C) |
| 2. Type Safety | 9.9 / 10 | 0 | 0 | 0 | 1 | New verifier helper and its regression tests use bare dict/tuple annotations and an untyped ModuleType |
| 3. Test Health | 9.7 / 10 | 0 | 0 | 0 | 3 | The binary-reward gate (pytest run -> test.sh reward) is never exercised; the module-scoped fixtures this PR refactored are bypassed |
| 4. Security | 9.9 / 10 | 0 | 0 | 0 | 1 | Task image still installs PyPDF2==3.0.1 (CVE-2023-36464, no fix available); nothing in-tree imports it |
| 5. Architecture & Design | 9.5 / 10 | 0 | 0 | 1 | 0 | CE071 hand-rolls its except-handler check and does not flag except Exception: pass, the shape this PR removed; its docstring and failure message do not state this gap |
| 6. Error Handling & Resilience | 9.5 / 10 | 0 | 0 | 1 | 0 | The patched court-form verifier still has silent except Exception: pass swallows; the one at line 121 swallows ET.ParseError on the shipped SC-100, so XFA checkbox extraction is always disabled with no signal |
| 7. API Surface & Maintainability | 9.8 / 10 | 0 | 0 | 0 | 2 | Exact == pin of pypdf added to the published dev extra only to mirror one sample's Dockerfile |
| 8. Evaluation Harness Quality | 9.9 / 10 | 0 | 0 | 0 | 1 | Grading contract change not stated: a flattened but visually correct SC-100 used to score 1.0 and now scores 0 |
Overall Score: 9.6 / 10 · Weakest Axis: Code Quality & Style at 8.7 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 4 · 🔵 11 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
- [Axis 1] CE071
_offendersis one 3-clause boolean comprehension with cyclomatic complexity 17 (radon C) (tests/test_custom_lint.py:5413) — radon cc givesM 5413:4 TestCE071PatchedBinaryVerifiersDoNotSkip._offenders - C (17). The method returns one list comprehension,return [ node.lineno for node in ast.walk(ast.parse(source)) if ( isinstance(node, ast.Call) and ... node.func.attr in {"skip", "importorskip", "xfail"} ) or ( isinstance(node, ast.Attribute) and node.attr in {"skip", "skipif", "xfail"} ... ) or ( isinstance(node, ast.ExceptHandler) and (node.type is None or ..."BaseException") ) ]. That puts three unrelated predicates (a pytest outcome call, a pytest.mark attribute, a broad except handler) in one expression of about 25 lines. This is the CC 10-20 band outside hot modules. Split it into three named module-level predicates (_is_pytest_outcome_call,_is_pytest_skip_mark,_is_broad_handler) and doreturn [n.lineno for n in ast.walk(tree) if any(p(n) for p in _PREDICATES)]. Each predicate then gets to CC 4 or less and can be read and tested on its own. ce021_guarded_evaluationresult_parse.py already has a broad-handler predicate (_GUARD_TYPES = frozenset({"ValueError", "Exception", "BaseException"})). Look at it before you write a third copy of that shape. - [Axis 1] CE071 scope is a hardcoded slug list; a third patched verifier would not be guarded, although the 'Modified from' marker could derive the scope (
tests/test_custom_lint.py:5440) —@pytest.mark.parametrize("slug", ["3d-scan-calc", "court-form-filling"])hardcodes the patched set. Each patched verifier already carries a machine-readable marker on line 1: court-form-filling/verifier/test_outputs.py:1 has# Modified from benchflow-ai/skillsbench@95b3e60: ...and 3d-scan-calc has# Modified from benchflow-ai/skillsbench@95b3e60: grade against fixed EXPECTED_* (issue #233)..grep -l 'Modified from benchflow'returns exactly these two files. The patched set is now listed in three places: this slug list, the class docstring (Scope: the locally patched 3d-scan-calc and court-form-filling verifiers.), and README.md plus .claude/notes/lint-rules.md. When someone patches a third verifier and adds the header, CE071 does not cover it. Derive the parametrization fromsorted(p for p in (REPO/'tasks/samples/skillsbench').glob('*/verifier/test_outputs.py') if p.read_text().startswith('# Modified from benchflow-ai/skillsbench')). The unpatched dialogue-parser verifier, which still haspytest.skipat lines 178 and 240, stays out because it has no marker. Add one assertion that the derived set is not empty. Change the docstring to name the marker and not the slugs. - [Axis 5] CE071 hand-rolls its except-handler check and does not flag
except Exception: pass, the shape this PR removed; its docstring and failure message do not state this gap (tests/test_custom_lint.py:5435) — CE071 builds its own except-handler detector inline:isinstance(node, ast.ExceptHandler) and (node.type is None or (isinstance(node.type, ast.Name) and node.type.id == "BaseException"))(lines 5435-5436). The repo already hastests/lint/rules/no_silent_except.py(CE005, wired intests/lint/runner.pyline 73), which flags bareexcept:ANDexcept Exception:handlers that neither re-raise nor log. CE071's copy is weaker. The defect this PR removed fromextract_checkbox_valueswasexcept Exception:\n pass, and CE071 does not flag that shape, so the guard written for issue #234 would not stop the same swallow from coming back. The patched verifier still has two such handlers attest_outputs.py:46and:121(except Exception:/pass), and CE071 accepts both. This also drifts from the documented placement: CLAUDE.md puts per-AST rules intests/lint/rules/(with a module docstring and# noqasupport through_is_suppressed) and keepstest_custom_lint.pyclasses for doc-surface and whole-tree rules. CE071 is a per-file AST rule, but it is written inline and does not honour# noqa: CE071, even thoughCE071was added to[tool.ruff.lint].external. Fix: move the skip/xfail half totests/lint/rules/ce071_*.pyas aBaseRule. Drop the hand-written except branch. Runcheck_paths(verifier_paths, rules=[Ce071Rule, NoSilentExcept])over the patched verifiers. Then either fix the two remaining XFA handlers or give each one a justified# noqa: CE005. - [Axis 6] The patched court-form verifier still has silent
except Exception: passswallows; the one at line 121 swallows ET.ParseError on the shipped SC-100, so XFA checkbox extraction is always disabled with no signal (tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py:121) — Two handlers on the XFA path still catch everything and do nothing. One is in parse_xfa_checkboxes at lines 121-122:except Exception:/pass. The other is in extract_xfa_xml at lines 46-47:except Exception:/pass. These are the CodeQL py/empty-except alerts @123 and @45 from main, now at lines 121 and 46 after the PR. I ran the verifier on environment/sc100-blank.pdf. extract_xfa_xml returns 4332 chars of datasets XML. The namespace-strip regexes on lines 97-99 remove only element prefixes and leave attribute prefixes like<FormName xfa:dataNode="dataGroup", soET.fromstringraisesParseError: unbound prefix: line 5, column 1every time. The line-121 handler swallows that error and parse_xfa_checkboxes returns {}. So 'Method 1: Extract from XFA XML' (line 61) has never worked on this form, even though the module and function docstrings (lines 8-10, 54) promise it. The PR removed the skip at the old line ~424, whose message said 'XFA extraction may not be supported'. Failing on missing checkbox data is the correct change, but a broken parser now turns into a hard failure whose message is justAvailable checkboxes: [], with nothing about the parse error. An agent that writes the XFA datasets instead of the AcroForm /V values gets a 0 that is hard to diagnose. There is no false pass: required-checkbox tests fail on missing data, and test.sh gives reward 1 only when pytest exits 0. That is why this is Medium and not High. Fix: (a) catch onlyET.ParseErrorand either fix the cleanup (also strip\s[a-zA-Z0-9_]+:[a-zA-Z0-9_]+="[^"]*"attribute prefixes) or re-raise, and narrow extract_xfa_xml's handler to KeyError/PdfReadError; (b) add an assertion in tests/test_skillsbench_court_form_verifier.py thatparse_xfa_checkboxes(extract_xfa_xml(_BLANK_PDF))parses without error, so the dead fallback cannot come back unnoticed. Note: test_checkbox_extraction_preserves_pdf_error passes only because Method 2 opens the PDF again (line 68). The PdfReadError from the first open is still swallowed at line 46.
Nits
- [Axis 1] The verifier regression test double-guards the no-skip rule, has a self-test that tests only the monkeypatch, and drifts from the existing skillsbench test pattern (
tests/test_skillsbench_court_form_verifier.py:25) — Lines 25-33 add an autouse fixturemonkeypatch.setattr(pytest, "skip", pytest.fail)and alsodef test_verifier_skip_fails(): with pytest.raises(pytest.fail.Exception, match="missing"): pytest.skip("missing"). The test only checks that the fixture's own monkeypatch works. The PR removes everypytest.skipfrom the verifier, and CE071 forbids them statically, so this fixture cannot trigger. Also, a skip during these direct calls would show as SKIPPED, not as a silent pass. That makes this a second mechanism for an invariant CE071 already enforces. Delete the fixture andtest_verifier_skip_fails, and let CE071 be the single guard. - [Axis 1] The patch header line is 169 characters, beyond the repo's 120 limit (ruff does not check this path) (
tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py:1) —# Modified from benchflow-ai/skillsbench@95b3e60: require form fields, preserve AcroForm read and field dereference errors, and expose extract_field_values (issue #234).is 169 characters. The sibling 3d-scan-calc header is one short line, and tasks/samples is in ruffextend-exclude, so nothing catches this. Make it as short as the sibling (for example# Modified from benchflow-ai/skillsbench@95b3e60: fail on missing form data (issue #234).) and leave the detail in README.md, which already lists it. - [Axis 1] uv.lock
revisiongoes down from 5 to 3 (lockfile written by an older uv) (uv.lock:2) — The diff changes-revision = 5to+revision = 3, andgit show origin/main:uv.lockhasrevision = 5. The PR only needed to add thepypdf7.0.1[crypto]entries. The header downgrade happened because the lock was regenerated with an older uv, and the next contributor on a current uv will flip it back. Regenerate the lock with the uv version that main uses, so the only diff is the pypdf additions. - [Axis 2] New verifier helper and its regression tests use bare
dict/tupleannotations and an untyped ModuleType (tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py:341) — The patch addsdef extract_field_values(pdf_path: Path) -> dict:with the docstring"""Return every AcroForm field value as a string ("" when unset).""". The new regression test file calls this helper directly. The baredictdoes not record the str -> str contract thatTestEmptyFields.test_field_emptyrelies on (len(value) < 3,value in ["", "None"]). Change the annotation to-> dict[str, str]and annotate the local accumulator asfield_values: dict[str, str] = {}. This file is vendored and ruff-excluded, and pyright does not check it, so the impact is only documentation. It is a cold path with no runtime defect: every truthy value goes throughstr(...)and every falsy value becomes"". - [Axis 3] The binary-reward gate (pytest run -> test.sh reward) is never exercised; the module-scoped fixtures this PR refactored are bypassed (
tests/test_skillsbench_court_form_verifier.py:37) — Each regression test calls a verifier method directly with a hand-built dict, for example_VERIFIER.TestCheckboxes().test_checkbox_value({}, *case)(line 39). As a result, thecheckbox_dataandpdf_field_valuesfixtures never run. This PR changedpdf_field_valuestoreturn extract_field_values(OUTPUT_FILE)(verifier line 338). The path from pytest outcome to reward is also never run: test.sh doesif [ $PYTEST_EXIT_CODE -eq 0 ]; then echo 1 > /logs/verifier/reward.txt. That path is the defect issue #234 describes. CE071 statically blocks the known skip spellings, so this is defense in depth. Recommended fix: add onepytester(orsubprocess+pytest.main) test that runs the verifier module withOUTPUT_FILEmonkeypatched to the shipped blank PDF (-k 'Checkboxes or EmptyFields', to avoid needing pdftotext). Assertoutcomes['failed'] > 0andoutcomes.get('skipped', 0) == 0. Then run it against the round-trip PDF and expect zero failures in those classes. - [Axis 3] No regression test covers the XFA checkbox branch, although the shipped SC-100 contains XFA datasets and the branch still swallows every exception (
tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py:62) —xfa_xml = extract_xfa_xml(pdf_path)(line 62) is the first of the verifier's two checkbox sources. On the shippedsc100-blank.pdfit returns a 4332-character datasets stream (checked directly). It is still wrapped inexcept Exception:\n passat lines 46-47 and lines 121-122. The new tests only fill and read AcroForm/Vvalues (writer.update_page_form_field_values(...), test line 106), so no test runs Method 1. Because this PR turns empty checkbox data into a hard fail, add a test that injects a filled XFA datasets stream (correct values only, no AcroForm/V). Pin whether it passes or fails as the intended contract. The REQUIRED/UNCHECKED patterns embed[0](for exampleList5[0].Lia[0].Checkbox5cb), which dotted XFA paths never contain. So an XFA-only fill cannot match those patterns, and the unchecked checks pass trivially. A test would make that behaviour explicit rather than accidental. - [Axis 3] The pin-parity guard covers pypdf only; cryptography is also on the read path of the encrypted SC-100 but is not pinned, so it can drift (
tests/test_skillsbench_court_form_verifier.py:158) —dockerfile_pins = re.findall(r"\bpypdf==(\S+)", ...)compares only pypdf. Its docstring says the regression tests 'only vouch for the pypdf the verifier image installs'. The same reasoning applies to cryptography.PdfReader(sc100-blank.pdf).is_encryptedisTrue(checked), so decryption goes throughpypdf[crypto]. The Dockerfile pinscryptography==50.0.2. The dev side has only the floor"cryptography>=48.0.1"(pyproject.toml line 228), and uv.lock resolves it to 50.0.2 today. Auv lock --upgradewould change it silently. Extend the test to compare the Dockerfile'scryptography==pin with thecryptographyversion in uv.lock (parse with tomllib), or add an exact dev pin. - [Axis 4] Task image still installs PyPDF2==3.0.1 (CVE-2023-36464, no fix available); nothing in-tree imports it (
tasks/samples/skillsbench/court-form-filling/environment/Dockerfile:17) — The PR edits this pip install block to pick up security fixes (line 14pypdf==7.0.1 \, line 19cryptography==50.0.2 \), but it keeps line 17PyPDF2==3.0.1 \. PyPDF2 is deprecated and its final release has CVE-2023-36464 (an infinite loop in content-stream parsing), which has no fixed version. A grep of the PR tree finds no import of PyPDF2 in the verifier or anywhere else in tasks/samples/skillsbench/: the verifier imports onlyfrom pypdf import PdfReader(test_outputs.py:30, :67, :343). The risk is in the DoS class only. The library is reachable only if the agent chooses to call it on a PDF that it made itself, inside its own throwaway task container. The README (tasks/samples/skillsbench/README.md:20-22) records this as an accepted upstream carry-over. Option A: remove PyPDF2 from the image, because pypdf 7.x is its maintained successor and has the same API family. This is a change to the upstream toolkit, so record it in the README's patch list. Option B: keep PyPDF2 for fidelity with upstream, and record it as an accepted risk in an allowlist next to the pip-audit config, so that future dependency scans of task images do not raise it again. CVSS:3.1/AV:L/AC:H/PR:L/UI:R/S:U/C:N/I:N/A:L - [Axis 7] Exact
==pin of pypdf added to the publisheddevextra only to mirror one sample's Dockerfile (pyproject.toml:71) —"pypdf[crypto]==7.0.1",is the only exact pin in[project.optional-dependencies].dev; every other entry uses>=. This extra ships in the wheel metadata, sopip install coder-eval[dev]forces pypdf 7.0.1 on consumers. The exact pin is there only to satisfytest_dockerfile_pypdf_pin_matches_dev_pin, which couples the framework's dev environment to one vendored sample image. Each pypdf CVE bump now needs a lockstep edit of both files. That is enforced, but it is a small permanent tax. Add a trailing comment on the pin naming the Dockerfile parity test so the next bumper knows why it is exact. Or, if the project adopts PEP 735[dependency-groups], move the pin there so it stays out of published metadata. - [Axis 7] README's expected-score table keeps a court-form value it now calls stale and not comparable (
tasks/samples/skillsbench/README.md:101) — Line 101 keeps| \court-form-filling` | 1.00 |in the 'what to expect' table. Line 104 then adds: 'The court-form score above predates the verifier fix for issue #234 and the pypdf update. Those historical scores are not comparable with scores from the updated sample.' A reader comparing a new run against the table sees a reference value that the doc itself says is invalid. Replace the cell with a marker such asTBD (re-baseline after #234)`, or re-run and record the new value. Then delete the caveat paragraph. - [Axis 8] Grading contract change not stated: a flattened but visually correct SC-100 used to score 1.0 and now scores 0 (
tasks/samples/skillsbench/README.md:15) — The PR removes the upstreampytest.skip("No checkbox data could be extracted from PDF (XFA extraction may not be supported)")and addsassert field_name in pdf_field_values, f"Field '{description}' not found in PDF form"(test_outputs.py:476). After this change, any output without AcroForm fields fails all 6 required-checkbox tests and all 20 empty-field tests. That includes a correctly filled form that was flattened, for example with the fillpdf flatten helpers that ship in the task image. The same agent output scored 1.0 before (the checkbox tests skipped and missing fields defaulted to ""). This is intended: to the verifier, a flattened form looks the same as the text-only PDF in issue #234 (case 5). But the initial_prompt says nothing about keeping the form fields, and the README only says it 'fails when required checkbox data or fields that must remain blank are missing'. So an analyst who reads a 0 cannot tell that flattening alone causes it. Add one explicit README sentence, for example: 'The output must keep the SC-100 AcroForm fields; a flattened PDF scores 0 even when it renders correctly.' Keep the upstream prompt unchanged.
What's Missing
Parallel paths:
- 🔵 court-form-filling.yaml was not updated with the verifier. Its success_criteria comment still says "Run the skillsbench verifier verbatim". The sibling 3d-scan-calc.yaml was updated when its verifier was patched: it says "patched locally", cites #233 and carries the #238 tamperability caveat. Make court-form-filling.yaml state the #234 patch and the same #238 caveat, so the YAML, README.md and the line-1 'Modified from' header all agree. (trigger: tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py)
- 🔵 The third sample, dialogue-parser, still calls pytest.skip (verifier lines 178 and 240), and nothing records why CE071 does not cover it. Its test.sh computes a partial reward as ctrf passed/tests, so a skip lowers the score rather than giving a false pass. The court-form test.sh, by contrast, gives reward 1 when pytest exits 0. Put this reason in the CE071 docstring or in .claude/notes/lint-rules.md. If the scope is derived from the marker, also say that the exclusion depends on binary versus partial reward, not on which files are patched. (trigger: tests/test_custom_lint.py) (restates: Axis 1: CE071 scope is a hardcoded slug list)
Tests:
- 🔵 The CE071 self-tests include no
except Exception: passcase, either positive or negative. test_detects_silent_outcomes covers only bareexcept:andexcept BaseException:, and test_allows_failure_and_narrow_exception covers onlyexcept ValueError. So the self-tests do not pin the exact shape this PR removed from extract_checkbox_values. Add a case that states the intended verdict fortry: read()\nexcept Exception: pass. (trigger: tests/test_custom_lint.py) (restates: Axis 5: CE071 hand-rolls its except-handler check and does not flagexcept Exception: pass) - 🔵 No test checks that the court-form verifier, when run as a whole through test.sh's pytest invocation, gives reward 0 for the shipped blank PDF and reward 1 for a correct round-trip PDF. Each new test calls a verifier method directly, so the outcome-to-reward path from issue #234 is never run end to end. (trigger: tests/test_skillsbench_court_form_verifier.py) (restates: Axis 3: The binary-reward gate (pytest run -> test.sh reward) is never exercised)
Downstream consumers:
- 🔵 The court-form scoring formula changed in three ways. A missing checkbox now fails where it used to skip. A missing blank field now fails where it used to default to "". The County[0].County[0] entry was replaced by TrialDateCourtAdd3[0]. So reward 1.0 no longer means the same thing as before. The README 'Typical score' table still shows 1.00, and earlier runs of this sample in runs/ or on the evalboard are not comparable with new ones. Re-baseline the table, or mark it TBD. (trigger: tasks/samples/skillsbench/README.md) (restates: Axis 7: README's expected-score table keeps a court-form value it now calls stale and not comparable)
- 🔵 pypdf is pinned in two places, the Dockerfile and the dev extra, and a parity test now keeps them equal. Cryptography is on the same decrypt path but has the same two places and no parity test: the Dockerfile pins ==50.0.2, while pyproject.toml line 228 has only the floor >=48.0.1. Extend the parity check to cryptography as well. (trigger: pyproject.toml) (restates: Axis 3: The pin-parity guard covers pypdf only; cryptography is also on the read path)
Daily/nightly:
- 🔵 The PR does not state its blast radius. The skillsbench samples are not in any scheduled workflow (no match in .github/workflows), so the grading change reaches only manual sample runs. The new exact
pypdf[crypto]==7.0.1in the dev extra has a wider reach. Every CI job that runsuv sync --frozen --extra devgets it (pr-checks.yml lines 96, 408, 559 and 758; harbor-e2e.yml line 86), and it ships in the published wheel metadata. The uv.lock revision downgrade from 5 to 3 is part of the same change. Say this in the PR description. (trigger: pyproject.toml) (restates: Axis 7: Exact==pin of pypdf added to the publisheddevextra only to mirror one sample's Dockerfile)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] Move CE071 out of tests/test_custom_lint.py (TestCE071PatchedBinaryVerifiersDoNotSkip, around line 5405) into tests/lint/rules/ce071_patched_verifier_no_skip.py as a BaseRule, and wire it in tests/lint/runner.py. Split the rule into named predicates:
_is_pytest_outcome_call(pytest.skip/importorskip/xfail) and_is_pytest_skip_mark(mark.skip/skipif/xfail). Delete the hand-written ExceptHandler branch and let CE005 (no_silent_except.py) handle broad handlers. Build the scope from the line-1 marker# Modified from benchflow-ai/skillsbench@: glob tasks/samples/skillsbench/*/verifier/test_outputs.py, keep only the files that start with the marker, and assert that the result is not empty. Then runcheck_paths(derived_paths, rules=[Ce071Rule, NoSilentExcept]), as test_custom_lint.py:491 already does. The rule docstring names the marker, not the slugs. A handler that stays on purpose needs a justified# noqa: CE005. Prevents: Prevents 4 findings. (1) The hardcoded slug list at test_custom_lint.py:5440: a third patched verifier would not be checked. (2) CE071 does not flagexcept Exception: pass, the exact shape that issue #234 removed, and has no# noqa: CE071path (test_custom_lint.py:5435). (3) The two swallows that stay in court-form-filling/verifier/test_outputs.py:46 and :121, which CE005 flags at once on the derived scope. (4) The radon C(17)_offenderscomprehension: the split into named predicates removes it. - [ce-lint] New whole-tree lint (a @pytest.mark.lint class in tests/test_custom_lint.py; next free number CE072). Every code in
[tool.ruff.lint].externalthat is NOT a doc-surface or whole-tree rule must be a BaseRule registered in tests/lint/runner.py, so_is_suppressedhonours# noqa: CEnnnfor it. To list the doc-surface or whole-tree classes that are allowed to live in test_custom_lint.py, use an explicit allowlist (WHOLE_TREE_RULES = {code: reason}). A new CE code that is in neither the registry nor the allowlist fails the build. Prevents: Prevents CE071 drifting from the CLAUDE.md placement rule. CE071 is a per-file AST rule. It went into test_custom_lint.py and was added to ruffexternal, but nothing honours# noqa: CE071(the finding at test_custom_lint.py:5435). - [ce-lint] Use the same marker-derived scope as the CE071 refactor. Add a lint test that runs
ruff check --isolated --line-length 120 --select E501,UP006,UP035on each patched verifier, that is, each tasks/samples file whose line 1 is# Modified from benchflow-ai/skillsbench@. Vendored files that nobody patched stay excluded (extend-exclude = ["tasks/samples"]stays). Only the lines this repo owns get the repo style gate. Prevents: Prevents the 169-character patch header at court-form-filling/verifier/test_outputs.py:1, which no check caught because tasks/samples is ruff-excluded. It also prevents more style drift in the locally patched verifiers. - [ce-lint] Generalize
test_dockerfile_pypdf_pin_matches_dev_pin(tests/test_skillsbench_court_form_verifier.py:152) into a static parity check. For everypkg==verpin in court-form-filling/environment/Dockerfile that is on the verifier's read path (pypdf AND cryptography), compare it with the version resolved in uv.lock (parse with tomllib), not with an exact pin in pyproject. Then change the dev extra back topypdf[crypto]>=7.0.1. This is delete-before-guard: the published==pin is only there because the test reads pyproject. Prevents: Prevents 2 findings. (1) cryptography is not covered by pin parity (test_skillsbench_court_form_verifier.py:158, pyproject.toml:226): auv lock --upgradechanges it silently. (2) The exactpypdf[crypto]==7.0.1pin in the publisheddevextra (pyproject.toml:71), which forces pypdf 7.0.1 oncoder-eval[dev]consumers. - [ce-lint] Add
required-version = ">=<uv version main uses>"to the existing[tool.uv]table in pyproject.toml. An older uv then refuses to write the lockfile. As a backstop, add a lint test that fails when therevisioninteger at the top of uv.lock is lower than the value on origin/main. A simpler form is to assertrevision >= 5. Prevents: Prevents the uv.lockrevision = 5->revision = 3downgrade (uv.lock:2), which happened because an older uv regenerated the lock. CI only runsuv sync --frozen, so nothing caught it. - [bandit-codeql] Add a
make audit-task-imagestarget, run from the security CI job. It takes thepkg==verpins from eachtasks/samples/**/environment/Dockerfilepip install block, writes them to tmp/ as a requirements file, and runspip-audit -r. The same table that the README's accepted carry-overs list feeds an allowlist (--ignore-vuln CVE-2023-36464for upstream PyPDF2==3.0.1, with a reason). Then the audit fails only on new CVEs. Prevents: Prevents task-image CVEs from going unrecorded: PyPDF2==3.0.1 (CVE-2023-36464, no fix) stayed at court-form-filling/environment/Dockerfile:17 while the PR bumped pypdf and cryptography for security fixes, and nothing scans the task images today.
Harness improvements (not statically reachable):
- Add one end-to-end verifier-outcome test with
pytester, or a subprocess runningpytest.main. Run court-form-filling/verifier/test_outputs.py with OUTPUT_FILE monkeypatched to the shipped sc100-blank.pdf (-k 'Checkboxes or EmptyFields', so pdftotext is not needed), and assertoutcomes['failed'] > 0andoutcomes.get('skipped', 0) == 0. Then run it on the round-trip filled PDF and assert zero failures in those classes. This test exercises thecheckbox_data/pdf_field_valuesfixtures and the pytest-exit-code -> test.sh reward gate. When it lands, delete the autousemonkeypatch.setattr(pytest, 'skip', pytest.fail)fixture andtest_verifier_skip_fails(test_skillsbench_court_form_verifier.py:25-33): the static rule plus this outcome test replace them. Why not static: The defect is in runtime behaviour: which tests skip or fail depends on what pypdf actually reads from a real (encrypted) PDF and on how the module-scoped fixtures resolve. An AST rule can forbid the known skip spellings (CE071), but it cannot prove that the gate turns a gap into a failure. Prevents: Prevents 2 findings: the binary-reward gate is never exercised and the refactored fixtures are bypassed (test_skillsbench_court_form_verifier.py:37), and the redundant self-testing skip guard (test_skillsbench_court_form_verifier.py:25). - Add an XFA smoke assertion to tests/test_skillsbench_court_form_verifier.py:
parse_xfa_checkboxes(extract_xfa_xml(_BLANK_PDF))must parse without ET.ParseError. Also add one test with a filled XFA datasets stream and no AcroForm /V values, which pins whether an XFA-only fill passes or fails as the intended contract. Why not static: The bug is data-dependent. The namespace-strip regexes leavexfa:dataNodeattribute prefixes, so ET.fromstring fails only on the real 4332-character datasets stream of the shipped SC-100. Finding this needs the PDF bytes and a parse at runtime, and the static CE005 can only flag the handler that hides the failure. Prevents: Prevents 2 findings: the silent swallow at test_outputs.py:121 that always disables XFA checkbox extraction on the shipped form, and the untested XFA branch at test_outputs.py:62. - Add a verifier-change checklist item to the SkillsBench README patch list, or to the PR template for tasks/samples. When a patched verifier changes its grading, (a) re-run the sample and update the expected-score row, or set it to
TBD (re-baseline after #NNN), and do not keep a value the doc calls stale; (b) add one sentence that states the new output contract (for example: 'the output must keep the SC-100 AcroForm fields; a flattened PDF scores 0'). Why not static: Whether a verifier change alters the grading contract, and what the new expected score is, needs semantic judgment and a live agent run. A digest-pinned table is possible, but it would only detect that something changed, not whether the value is still right. Prevents: Prevents 2 findings: the README expected-score table keeps a court-form value that it calls stale (tasks/samples/skillsbench/README.md:101), and the unstated grading-contract change for flattened PDFs (README.md:15).
Top 5 Priority Actions
- Fix the XFA checkbox path in tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py:121 (and :46): the namespace cleanup at lines 97-99 leaves attribute prefixes such as
xfa:dataNode, soET.fromstringalways raises ParseError on the shipped SC-100 andexcept Exception: passhides it; an agent that fills only the XFA datasets scores 0 with the messageAvailable checkboxes: []; catch onlyET.ParseError, also strip attribute prefixes or re-raise, and add a test thatparse_xfa_checkboxes(extract_xfa_xml(_BLANK_PDF))parses. - State the new grading contract in tasks/samples/skillsbench/README.md:15: a flattened SC-100 that renders correctly scored 1.0 before and now fails 26 tests (score 0) for the same agent output, because the skip is gone and
assert field_name in pdf_field_values(test_outputs.py:476) is new; add one sentence that the output must keep the AcroForm fields, and change the stalecourt-form-filling | 1.00row at README.md:101 to TBD until you re-baseline it. - Close the CE071 gap at tests/test_custom_lint.py:5435: its own except-handler check does not flag
except Exception: pass, which is the swallow this PR removed, so it accepts the two that remain; move the skip/xfail check to aBaseRulein tests/lint/rules/ with# noqa: CE071support, delete the inline handler branch, and run it together with CE005 (no_silent_except) on the patched verifiers. - Get CE071's scope from the marker, not from the hardcoded slug list at tests/test_custom_lint.py:5440: glob skillsbench
*/verifier/test_outputs.pyfiles whose line 1 starts with# Modified from benchflow-ai/skillsbench, assert the set is not empty, and make the docstring name the marker; while you are there, split the CC-17_offenderscomprehension at :5413 into three named predicates (the broad-handler one must match only a bareexceptorexcept BaseException). - Make the verifier's tests cover the real reward path and every dependency on it: in tests/test_skillsbench_court_form_verifier.py:158, extend the pin-parity check to compare the Dockerfile's
cryptography==50.0.2with uv.lock (the SC-100 is encrypted); add one pytester run of the verifier module on the blank PDF that asserts failed > 0 and skipped == 0; delete the redundant autouse skip-to-fail fixture at :25; and regenerate uv.lock with main's uv sorevisionstays at 5.
Stats: 0 🔴 · 0 🟠 · 4 🟡 · 11 🔵 across 8 axes reviewed.
bai-uipath
left a comment
There was a problem hiding this comment.
Ran the old and new verifier through pytest (-k "Checkboxes or EmptyFields", pypdf 7.0.1) against four PDFs:
| main | this PR | |
|---|---|---|
| Blank form copied unchanged | 20 pass, 12 skip | 6 fail |
| No form fields | 20 pass, 12 skip | 26 fail |
| Text filled, no box ticked | 20 pass, 12 skip | 6 fail |
| Correct AcroForm fill | 32 pass | 32 pass |
The PR closes the path where skipped tests still earned reward 1, and a correct fill still passes.
Two things before merge:
uv.lockrevisiondrops from 5 to 3. Regenerate it with the uv version main uses, so the only diff is the pypdf entries.pypdf[crypto]==7.0.1in thedevextra meanspip-auditandosv-scannercheck pypdf on every PR. pypdf has had 23 advisories since Oct 21, so each new one will fail the security step on unrelated PRs until three files are bumped together. Suggest>=7.0.1, with the parity test comparing the Dockerfile pin to the version inuv.lock. A future fix is thenuv lock -P pypdfplus the Dockerfile line.
Non-blocking: I confirmed the XFA checkbox parse fails on the shipped SC-100 (ParseError: unbound prefix), and except Exception: pass hides the error. That was already broken before this PR, so a follow-up is fine.
- Relax the dev pin to pypdf[crypto]>=7.0.1 and check the Dockerfile pypdf and cryptography pins against uv.lock. Regenerate uv.lock with a current uv so revision stays at 5. - Replace the verifier's silent XFA handlers with narrow handlers that warn. Keep upstream's AcroForm-only checkbox grading and pin it with a test, because parsing the XFA datasets would change grading. - Move CE071 to a BaseRule, take its scope from the "Modified from" marker, and run it with CE005 over the patched verifiers. - Add whole-verifier pytest runs for a blank form and a correct fill, and remove the redundant skip-to-fail fixture. - Document the AcroForm output contract, mark the court-form score TBD, and cite #234 and #238 in the task YAML. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reply to the 8-axis review. Commit 4502f3a fixes most findings; the items that are not fixed give the reason. Non-blocking
Nits
What's Missing
Not done in this PR (scope)These items are good ideas, but they are outside this fix, and they can be follow-ups: CE072 (registry or allowlist placement), ruff on the patched verifiers, VerificationAll |
|
@bai-uipath Thank you for the old-vs-new run. 4502f3a fixes both items:
XFA: the handlers are now narrow and emit a warning, so the failure is no longer silent. I tried to fix the parse, but it changes grading (details in my reply to the other review), so I agree that the parser fix should be a follow-up. |

Supersedes #243 by @gezerui888, which this branch carries unchanged with its commit authorship. It adds one follow-up commit and a merge of current
main. Fixes #234.The fix from #243
The
court-form-fillingverifier gave reward 1.0 to a text-only PDF with no form fields, and to a form with no checkboxes ticked. In those cases all 12 checkbox tests skipped, andtest.shcounts a pytest exit 0 as a pass. The fix:pytest.skippaths. Empty checkbox data now fails.TestEmptyFieldsfails when a field is missing. Before, a missing field counted as empty.Follow-up commit
uv.lock). 6.21.4 has five GHSA advisories that are fixed only in 7.0.0 or 7.0.1: GHSA-cmpx-464x-82c3 (XFA memory use; the verifier parses XFA from the agent's PDF), GHSA-rfv5-695j-j993, GHSA-fhff-j4m8-439h, GHSA-59jv-x5x2-gg4c and GHSA-mwj3-fpmc-hjcv. The PyPI/OSV feeds do not list them yet, sopip-auditdoes not flag them now. It will fail when the feeds catch up.PyPDF2==3.0.1stays from upstream. It is the final release of that package, and CVE-2023-36464 has no fix in it.extract_field_values(pdf_path), and the tests call it andextract_checkbox_valuesdirectly instead offixture.__wrapped__.Verification
make verify: 6,972 passed, 2 skipped, 93.42% coverage. Lint, format and typecheck are clean.🤖 Generated with Claude Code