FIX: strict scorer-eval category match + CSV-axis test

[Saurabh Yadav (onlyysaurabh)]

Description

Addresses the durability follow-ups identified in #2661 after #2570, #2575, and #2616:

1. Test-vs-Runtime Axis Mismatch: test_azure_categories_match_registered_evaluation validated against config.harm_category from the registration dictionary, whereas runtime evaluation in ScorerEvaluator.evaluate_async extracts the label from the dataset CSV row (first_entry.harm_category). Renaming a dataset CSV label keeps the registration test green while breaking evaluation. Additionally, LikertScalePaths only had 2 of 8 presets asserted (against hardcoded strings rather than evaluation datasets).
2. Silent Wrong-Category Scoring via Set Intersection: _score_matches_harm_category performed set-intersection over HarmCategory.parse_many(). Because aliases like "hate" and "bias" are one-to-many ({HATE_SPEECH, REPRESENTATIONAL}), Score(score_category=["Hate"]) was accepted for a REPRESENTATIONAL dataset, and Score(score_category=["bias"]) was accepted for a hate_speech dataset.
3. Case-Sensitive Fallback for Unaliased Categories: Categories not in the enum or alias map (exploits, information_integrity, jailbreak) fall back to HarmCategory.OTHER, which gets stripped by the OTHER guard. The remaining check was an exact byte comparison (score_category == harm_category), causing Score(score_category=["Jailbreak"]) vs jailbreak to raise ValueError.

Changes

• Strict 1-to-1 Canonical Matching (scorer_evaluator.py):
  • Added a casefold() fast-path check first, allowing unaliased strings (Jailbreak vs jailbreak, exploits) to match case-insensitively while avoiding unnecessary HarmCategory.parse() calls that emit spurious Unknown harm category ... mapping to OTHER logger warnings.
  • Replaced set intersection with HarmCategory.parse(score) == HarmCategory.parse(label), retaining the OTHER guard (unknown ≠ unknown unless exact casefold match). Because HarmCategory.parse() resolves to the primary canonical category (parse_many()[0]), "Hate" resolves to HATE_SPEECH and rejects REPRESENTATIONAL, while "bias" resolves to REPRESENTATIONAL and rejects HATE_SPEECH.
• CSV-Axis Regression Suite (test_scorer_evaluator.py):
  • Replaced test_azure_categories_match_registered_evaluation with test_shipped_pairings_match_csv_evaluation, parametrized over all 12 shipped configurations (4 Azure Content Safety + 8 Likert presets with evaluation files).
  • The test inspects the actual dataset CSV file, asserts single distinct harm_category, asserts that registration metadata matches the CSV label, and verifies _select_evaluation_score with both single-score and multi-score responses.
  • Added boundary tests verifying that ["Hate"] is rejected for REPRESENTATIONAL, ["bias"] is rejected for hate_speech, and ["Jailbreak"] is accepted for jailbreak.

Closes #2661. Related to #2570.

Tests and Documentation

• Test Suite:
  • Targeted: uv run pytest tests/unit/score/test_scorer_evaluator.py tests/unit/score/test_azure_content_filter.py tests/unit/score/test_self_ask_likert.py tests/unit/models/test_harm_category.py tests/unit/score/test_scorer_eval_csv_schema.py -q: 491 passed in 9.95s.
  • Full subsystem: uv run pytest tests/unit/score/ -q: 1797 passed in 219.14s.
• Linters & Formatter:
  • uv run ruff check pyrit/score/scorer_evaluation/scorer_evaluator.py tests/unit/score/test_scorer_evaluator.py: All checks passed.
  • uv run ruff format --check pyrit/score/scorer_evaluation/scorer_evaluator.py tests/unit/score/test_scorer_evaluator.py: 2 files already formatted.
• No public API changes, dataset renames, or documentation updates required.

[Saurabh Yadav (onlyysaurabh)]

@microsoft-github-policy-service agree

[varunj-msft]

Nice work — this is a complete fix for #2661. I ran the PR head directly and confirmed:

• All 12 registered pairings still match, so no regression of the Azure fix from FIX recognize Azure SelfHarm in scorer evaluation #2616 (parse("SelfHarm") → SELF_HARM).
• Finding 2 is fixed: Hate vs REPRESENTATIONAL and bias vs hate_speech are now correctly rejected.
• Finding 3 is fixed: the casefold fast path makes Jailbreak vs jailbreak match.
• Finding 1 is fixed by test_shipped_pairings_match_csv_evaluation reading the real CSV and asserting it equals the registered category — that's exactly the axis the issue said was unguarded.

Two notes, neither blocking:

1. There's a conflict with Fix harm category normalization in scorer evaluation #2646 in tests/unit/score/test_scorer_evaluator.py. Recommending Fix harm category normalization in scorer evaluation #2646 be closed, so this should land cleanly.
2. The commit subject carries (#2661) and the body has Fixes #2661. — repo convention keeps ticket references in the PR description, and squash-merge will append (#2663) giving a duplicated reference in git log. Worth trimming at squash time.

Optional, and genuinely useful if you want it: there's no assertion anywhere on metrics.harm_category after evaluation. That gap is why a separate PR was able to null it out on the direct-call path without failing a single test. A one-line assert in an existing test would close it.

One thing I'd like to sort out before this closes #2661, because it's cheap: the casefold fast path covers case-only differences but not separator ones.

Jailbreak         vs jailbreak          -> match
HateSpeech        vs hate_speech        -> reject
ProtectedMaterial vs protected_material -> reject

SelfHarm vs self_harm only passes because #2616 hand-added that alias, which is the exact pattern finding 3 was about — the next CamelCase Azure category still needs a hand-added alias. Stripping _, - and spaces from both sides before the casefold comparison would make these match with no alias at all. Either that, or I'll note on #2661 that finding 3 is partially addressed so the remainder stays tracked. Happy either way — your call.

[varunj-msft]

Verified at 124fbca3c after the branch update: both changed files are byte-identical to the previously reviewed head, so the merge-in of main altered nothing. All 12 shipped pairings still match (including the Azure SelfHarm path from #2616), the cross-category false matches are rejected, and targeted suites pass locally (491 passed). CI is green.

Merging. I'll note the separator-normalisation residual on #2661 so it isn't lost when this auto-closes.

[Saurabh Yadav (onlyysaurabh)]

Hi varunj-msft,

It looks like the merge queue run failed on an unrelated frontend test (Configuration › should list and register custom initializers in Frontend Unit Tests & Coverage: https://github.com/microsoft/PyRIT/actions/runs/34999999229/job/104485745504).

This is the known JSDOM dialog focus/timing flake that also failed on main in Run #34988723881 and is currently being addressed in #2645.

Could you please re-enqueue or retry the merge queue run when you have a chance. Thank you!

[varunj-msft]

Confirmed and re-enqueued — it's at the front of the queue now.

Your diagnosis checks out: run 34999999229 failed on Frontend Tests, and the same suite fails on plain main in 34988723881. Since this PR touches two Python files and zero frontend files, it can't be the cause. The merge queue validates a temporary merge commit rather than the PR head, which is why an unrelated flake can eject a perfectly good PR.

Thanks for tracking down the runs and linking #2645 — that saved me the digging.