Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .claude/notes/lint-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -961,3 +961,17 @@ the module as a parameter instead of a third copy.
Taking the class name from the model follows CE056 importing `IN_CONTAINER_ENV` and CE057
deriving its target set from `SIDECAR_MODULES`. Alias resolution removes the *local
binding* spelling, not every rename; taking the name from the model covers the rest.

## CE071

The court-form verifier skipped checks when checkbox data or blank-field keys
were absent. Its shell runner treated pytest's successful exit as reward 1.
CE071 rejects skip and xfail calls or marks in every SkillsBench verifier whose
first line carries the `# Modified from benchflow-ai/skillsbench@` marker, so a
newly patched verifier is in scope without a list edit. The same test runs CE005
on those files, because the removed defect was an `except Exception: pass` that
an inline `BaseException`-only check did not see. `dialogue-parser` is out of
scope: it is unpatched, and its runner scores passed / total, so a skip lowers
the reward instead of passing it. Real PDF regression tests, including two
whole-verifier pytest runs, anchor the court-form field names, extraction and
exit code to the shipped input.
6 changes: 4 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ Changelog = "https://github.com/UiPath/coder_eval/blob/main/CHANGELOG.md"

[project.optional-dependencies]
dev = [
"pypdf[crypto]>=7.0.1",
"pytest-asyncio>=1.3.0",
"pytest>=9.0.2",
"pytest-mock>=3.15.1",
Expand Down Expand Up @@ -242,8 +243,8 @@ constraint-dependencies = [
[tool.ruff]
line-length = 120
target-version = "py313"
# Vendored third-party sample tasks (benchflow-ai/skillsbench) are kept verbatim;
# they are not framework code and are not linted/formatted.
# Vendored third-party sample tasks (benchflow-ai/skillsbench) include local
# verifier patches documented in their README. Keep them outside framework formatting.
extend-exclude = ["tasks/samples"]

[tool.ruff.lint]
Expand Down Expand Up @@ -311,6 +312,7 @@ external = [
"CE065",
"CE066",
"CE068",
"CE071",
] # custom architectural lint rules (tests/lint/)

[tool.ruff.lint.pylint]
Expand Down
51 changes: 33 additions & 18 deletions tasks/samples/skillsbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,33 @@ evaluation (not just a smoke test) with one command.
They are vendored from [**benchflow-ai/skillsbench**](https://github.com/benchflow-ai/skillsbench)
(Apache-2.0), pinned at commit
[`95b3e60`](https://github.com/benchflow-ai/skillsbench/tree/95b3e60de7a075090e2f9137f32e0d24da1a289b).
Each sample keeps the upstream prompt, inputs, and verifier; only the container
base and the coder_eval task wiring were added. The one local change is to the
`3d-scan-calc` verifier, which is patched to grade against fixed expected
values instead of re-deriving them from the agent-writable `scan_data.stl`
(issue #233). The other verifiers are unchanged and can still read
agent-writable inputs: `dialogue-parser` takes part of its ground truth from
`/app/script.txt`.

That patch closes only the `scan_data.stl` vector. Grading still runs in the
agent's container, so the `pytest` on `PATH`, auto-loaded `pytest11` plugins,
site-packages, and the `$TASK_DIR` verifier copy remain tamperable by the
agent. See the follow-up issue
Each sample keeps the upstream prompt and inputs. The container base and
coder_eval task wiring were added locally, along with two verifier patches:

- `3d-scan-calc` grades against fixed expected values instead of re-deriving
them from the agent-writable `scan_data.stl` (issue #233).
- `court-form-filling` fails when required checkbox data or fields that must
remain blank are missing (issue #234). Its blank-field checks also replace
a nonexistent county field with the omitted third trial court address field.
AcroForm read and field dereference errors propagate to the test runner.
The output must keep the SC-100 AcroForm fields: a flattened PDF scores 0,
even when it renders correctly. Checkbox grading reads AcroForm values only.
The verifier's XFA fallback cannot parse the shipped form's datasets, as
upstream, and now emits a warning when that happens.
Its environment pins pypdf 7.0.1 (upstream 5.1.0) and cryptography 50.0.2
(upstream 43.0.3) to pick up their security fixes. It keeps upstream's
`PyPDF2==3.0.1`, the final release of that deprecated package, which has
known advisories with no fixed version.

The `dialogue-parser` verifier is unchanged and takes part of its ground truth
from the agent-writable `/app/script.txt`. The `court-form-filling` verifier
also reads the agent-writable `/root/sc100-blank.pdf` for its input-versus-output
identity check.

The `3d-scan-calc` patch closes only the `scan_data.stl` vector. Grading still
runs in the agent's container, so the `pytest` on `PATH`, auto-loaded `pytest11`
plugins, site-packages, and the `$TASK_DIR` verifier copy remain tamperable by
the agent. See the follow-up issue
[#238](https://github.com/UiPath/coder_eval/issues/238).

## What they do
Expand Down Expand Up @@ -56,8 +71,8 @@ coder-eval run tasks/samples/skillsbench/*/*.yaml

Each task builds its container, runs the agent (Claude Sonnet 4.6 by default) in
an isolated sandbox, then scores the result with the upstream verifier (for
`3d-scan-calc`, the locally patched one described above). Budget roughly 1–3
minutes per task.
`3d-scan-calc` and `court-form-filling`, the local patches described above).
Budget roughly 1–3 minutes per task.

## Interpreting the results

Expand Down Expand Up @@ -87,7 +102,7 @@ is non-deterministic, so exact numbers vary from run to run:
| Task | Typical score |
|------|---------------|
| `3d-scan-calc` | 1.00 |
| `court-form-filling` | 1.00 |
| `court-form-filling` | TBD (re-baseline after #234) |
| `dialogue-parser` | ~0.83 |

## How it works (brief)
Expand All @@ -101,6 +116,6 @@ is non-deterministic, so exact numbers vary from run to run:

These tasks are derived from benchflow-ai/skillsbench, licensed under Apache-2.0.
The prompts, input data, and verifiers are the work of their original authors
(credited in each upstream task's `task.md`). The `3d-scan-calc` verifier is
patched locally to grade against fixed expected values. See the pinned commit
above for the upstream source of truth.
(credited in each upstream task's `task.md`). The `3d-scan-calc` and
`court-form-filling` verifiers have the local patches described above. See the
pinned commit above for the upstream source of truth.
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,13 @@ run_limits:
turn_timeout: 300

success_criteria:
# Run the skillsbench verifier verbatim: copy it from the task dir into a
# writable /verifier, run its test.sh (writes /logs/verifier/reward.txt), and
# emit that reward (0.0-1.0) as the criterion score.
# Run the skillsbench verifier (patched locally to fail on missing checkbox
# data or form fields, #234): copy it from the task dir into a writable
# /verifier, run its test.sh (writes /logs/verifier/reward.txt), and emit
# that reward (0.0-1.0) as the criterion score.
# Grading still runs in the agent's container, so the pytest on PATH,
# auto-loaded pytest11 plugins, site-packages and the $TASK_DIR verifier
# copy remain tamperable; see the follow-up issue #238.
- type: run_command
command: |
set -e
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@ RUN apt-get update \
&& apt-get install -y --no-install-recommends poppler-utils \
&& rm -rf /var/lib/apt/lists/* \
&& pip install --no-cache-dir \
pypdf==5.1.0 \
pypdf==7.0.1 \
fillpdf==0.7.3 \
pdfrw==0.4 \
PyPDF2==3.0.1 \
reportlab==4.2.5 \
cryptography==43.0.3 \
cryptography==50.0.2 \
pytest==8.4.1 \
pytest-json-ctrf==0.3.5

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Modified from benchflow-ai/skillsbench@95b3e60: fail on missing form data (issue #234).
"""
Tests for court-form-filling task (SC-100 Small Claims Court form).

Expand All @@ -11,6 +12,7 @@

import re
import subprocess
import warnings
import xml.etree.ElementTree as ET
from pathlib import Path

Expand All @@ -25,8 +27,10 @@ def extract_xfa_xml(pdf_path: Path) -> str:
Extract XFA XML datasets from a PDF file.
Returns the raw XML string or empty string if not found.
"""
from pypdf import PdfReader
from pypdf.errors import PdfReadError

try:
from pypdf import PdfReader
reader = PdfReader(str(pdf_path))

# Try to access XFA data
Expand All @@ -42,8 +46,8 @@ def extract_xfa_xml(pdf_path: Path) -> str:
if stream_name == "datasets":
stream_obj = xfa[i + 1].get_object()
return stream_obj.get_data().decode("utf-8", errors="ignore")
except Exception:
pass
except (KeyError, PdfReadError) as exc:
warnings.warn(f"XFA datasets could not be read; using AcroForm checkboxes only: {exc!r}", stacklevel=2)
return ""


Expand All @@ -63,26 +67,23 @@ def extract_checkbox_values(pdf_path: Path) -> dict:
checkbox_values.update(parse_xfa_checkboxes(xfa_xml))

# Method 2: Try pypdf get_fields() for AcroForms
try:
from pypdf import PdfReader
reader = PdfReader(str(pdf_path))
fields = reader.get_fields() or {}

for name, field in fields.items():
# Check if it's a checkbox/radio field
if "Checkbox" in name or field.get("/FT") in ["/Btn"]:
value = field.get("/V", "")
if value:
if hasattr(value, "get_object"):
value = str(value.get_object())
else:
value = str(value)
# Normalize value
value = value.strip("/")
if value and value not in ["Off", "None", ""]:
checkbox_values[name] = value
except Exception:
pass
from pypdf import PdfReader
reader = PdfReader(str(pdf_path))
fields = reader.get_fields() or {}

for name, field in fields.items():
# Check if it's a checkbox/radio field
if "Checkbox" in name or field.get("/FT") in ["/Btn"]:
value = field.get("/V", "")
if value:
if hasattr(value, "get_object"):
value = str(value.get_object())
else:
value = str(value)
# Normalize value
value = value.strip("/")
if value and value not in ["Off", "None", ""]:
checkbox_values[name] = value

return checkbox_values

Expand Down Expand Up @@ -120,8 +121,8 @@ def find_checkboxes(element, path=""):
find_checkboxes(child, current_path)

find_checkboxes(root)
except Exception:
pass
except ET.ParseError as exc:
warnings.warn(f"XFA datasets did not parse; using AcroForm checkboxes only: {exc}", stacklevel=2)

return checkbox_values

Expand Down Expand Up @@ -268,7 +269,6 @@ def normalize_text(text: str) -> str:
("SC-100[0].Page1[0].CaptionRight[0].CN[0].CaseName[0]", "page1_case_name"),
("SC-100[0].Page1[0].CaptionRight[0].CN[0].CaseNumber[0]", "page1_case_number"),
("SC-100[0].Page1[0].CaptionRight[0].County[0].CourtInfo[0]", "page1_court_info"),
("SC-100[0].Page1[0].CaptionRight[0].County[0].County[0]", "page1_county"),
# Trial date section 1
("SC-100[0].Page1[0].Order[0].List1[0].LI1[0].TrialDate1[0]", "page1_trial_date1"),
("SC-100[0].Page1[0].Order[0].List1[0].LI1[0].TrialTime1[0]", "page1_trial_time1"),
Expand All @@ -283,6 +283,7 @@ def normalize_text(text: str) -> str:
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDate3[0]", "page1_trial_date3"),
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialTIme3[0]", "page1_trial_time3"),
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDepartment3[0]", "page1_trial_dept3"),
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateCourtAdd3[0]", "page1_court_addr3"),
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateClerkSign[0]", "page1_clerk_sign"),
("SC-100[0].Page1[0].Order[0].List1[0].LI3[0].TrialDateClerkSignDate[0]", "page1_clerk_date"),
# Second plaintiff/defendant (only one of each in this case)
Expand Down Expand Up @@ -334,23 +335,25 @@ def checkbox_data():
@pytest.fixture(scope="module")
def pdf_field_values():
"""Extract all field values from the filled PDF using pypdf."""
from pypdf import PdfReader

if not OUTPUT_FILE.exists():
pytest.fail(f"Output file not found at {OUTPUT_FILE}")

reader = PdfReader(str(OUTPUT_FILE))
return extract_field_values(OUTPUT_FILE)


def extract_field_values(pdf_path: Path) -> dict[str, str]:
"""Return every AcroForm field value as a string ("" when unset)."""
from pypdf import PdfReader

reader = PdfReader(str(pdf_path))
fields = reader.get_fields() or {}

field_values = {}
field_values: dict[str, str] = {}
for name, field in fields.items():
value = field.get("/V", "")
if value:
if hasattr(value, "get_object"):
try:
value = str(value.get_object())
except:
value = str(value)
value = str(value.get_object())
else:
value = str(value)
field_values[name] = value if value else ""
Expand Down Expand Up @@ -421,9 +424,6 @@ def test_checkbox_value(self, checkbox_data, field_pattern, description, expecte
}

if not matching_fields:
# If no checkboxes found at all, it might be extraction failure
if not checkbox_data:
pytest.skip("No checkbox data could be extracted from PDF (XFA extraction may not be supported)")
pytest.fail(
f"Checkbox '{description}' (pattern: {field_pattern}) not found. "
f"Available checkboxes: {list(checkbox_data.keys())}"
Expand Down Expand Up @@ -459,9 +459,6 @@ def test_checkbox_unchecked(self, checkbox_data, field_pattern, description):
}

if not matching_fields:
# If no checkboxes found at all, it might be extraction failure
if not checkbox_data:
pytest.skip("No checkbox data could be extracted from PDF")
# Field not found means it's likely unchecked (not in filled data)
return

Expand All @@ -479,6 +476,7 @@ class TestEmptyFields:
ids=[f[1] for f in EMPTY_FIELDS])
def test_field_empty(self, pdf_field_values, field_name, description):
"""Verify field is empty or has minimal placeholder."""
value = pdf_field_values.get(field_name, "")
assert field_name in pdf_field_values, f"Field '{description}' not found in PDF form"
value = pdf_field_values[field_name]
is_empty = not value or value in ["", "None"] or len(value) < 3
assert is_empty, f"Field '{description}' should be empty, got: '{value}'"
61 changes: 61 additions & 0 deletions tests/lint/rules/ce071_patched_verifier_no_skip.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""CE071: A locally patched binary-reward verifier must not skip or xfail.

Flags ``pytest.skip`` / ``pytest.importorskip`` / ``pytest.xfail`` calls and
``pytest.mark.skip`` / ``skipif`` / ``xfail`` marks. A binary verifier's runner
writes reward 1 when pytest exits 0, so a skipped check counts as a pass.

Scope: every ``tasks/samples/skillsbench/*/verifier/test_outputs.py`` whose first
line starts with ``PATCHED_MARKER``. The test runs this rule together with CE005,
which catches the broad ``except Exception: pass`` swallow. ``dialogue-parser`` is
out of scope because it is unpatched and its runner scores passed / total, so a
skip lowers its reward instead of passing it.

BLIND SPOT: aliased imports (``from pytest import skip``) and dynamically built
calls are not resolved.

Add ``# noqa: CE071`` on the offending line for a deliberate exception.

Rationale: .claude/notes/lint-rules.md § CE071
"""

import ast

from tests.lint.rules.base import BaseRule


PATCHED_MARKER = "# Modified from benchflow-ai/skillsbench@"
_OUTCOME_CALLS = frozenset({"skip", "importorskip", "xfail"})
_SKIP_MARKS = frozenset({"skip", "skipif", "xfail"})


def _is_pytest_name(node: ast.expr) -> bool:
return isinstance(node, ast.Name) and node.id == "pytest"


def _is_pytest_outcome_call(node: ast.Call) -> bool:
func = node.func
return isinstance(func, ast.Attribute) and func.attr in _OUTCOME_CALLS and _is_pytest_name(func.value)


def _is_pytest_skip_mark(node: ast.Attribute) -> bool:
owner = node.value
return (
node.attr in _SKIP_MARKS
and isinstance(owner, ast.Attribute)
and owner.attr == "mark"
and _is_pytest_name(owner.value)
)


class PatchedVerifierNoSkip(BaseRule):
id = "CE071"

def visit_Call(self, node: ast.Call) -> None:
if _is_pytest_outcome_call(node):
self.violation(node, "patched binary verifier must fail, not skip or xfail")
self.generic_visit(node)

def visit_Attribute(self, node: ast.Attribute) -> None:
if _is_pytest_skip_mark(node):
self.violation(node, "patched binary verifier must not carry a skip or xfail mark")
self.generic_visit(node)
Loading
Loading