Fix harm category normalization in scorer evaluation

[San (San-exe)]

Summary

Fix scorer evaluation when the dataset harm_category and scorer output category use different naming schemes for the same harm category.

Changes

• Pass the configured dataset evaluation harm_category through ScorerEvaluator.run_evaluation_async() into evaluate_dataset_async().
• Normalize evaluation score category matching against the configured evaluation category while preserving the existing rejection of unrelated categories.
• Add fairness_bias as an alias for REPRESENTATIONAL in HarmCategory.
• Add regression coverage for fairness_bias → REPRESENTATIONAL and sexual → SEXUAL_CONTENT category matching.

Validation

• 113 tests passed across the relevant scorer evaluator, Self-Ask Likert, and Azure Content Filter test suites.
• git diff --check is clean.

[San (San-exe)]

@microsoft-github-policy-service agree

[varunj-msft]

Thanks for picking this up.

Some context that isn't obvious from the outside: #2570 was already fixed by #2575 and #2616 before this PR landed, and the remaining gaps got tracked in #2661. #2663 has since taken the same ground, and because it tightens the matcher rather than changing which value is compared, it covers all three of the #2661 findings. The two PRs also conflict — both rewrite TestSelectEvaluationScore in tests/unit/score/test_scorer_evaluator.py, so whichever merges first breaks the other.

Two things worth knowing for future reference, because they're genuinely subtle and both would have been easy to miss in review:

1. The configured category can mask a genuinely mismatched dataset. Because the configured value replaces the human-label category unconditionally, there's no longer an equivalence check between them. Pairing a Sexual scorer output with harm/violence.csv and a configured SEXUAL_CONTENT is rejected on main today, but accepted here — and it writes a registry row with category=SEXUAL_CONTENT alongside harm_definition=violence.yaml. That guard exists specifically to catch a mis-paired dataset, so this inverts it.

2. Objective evaluation regresses. ScorerEvalDatasetFiles documents harm_category as "Required for harm evaluations, ignored for objective evaluations", but it's now forwarded for objective datasets too. A named true/false score (SelfAskRefusalScorer defaults to ["refusal"]) with a configured harm_category returns metrics on main and raises ValueError here.

Two smaller ones while you're in there:

3. The metrics lose their harm category on the public path. The CSV-derived local is renamed to dataset_harm_category, and the new harm_category parameter defaults to None — but _compute_metrics is still called with the parameter:

harm_category=harm_category,   # the new parameter, None unless passed

run_evaluation_async passes it, so the normal path is fine. A direct call to the public evaluate_dataset_async writes harm_category=None into the metrics registry.

4. harm_category is inserted as the second positional parameter of a public method, ahead of the existing ones. evaluate_dataset_async(dataset, 3) would now bind 3 to harm_category instead of num_scorer_trials — a wrong result rather than a TypeError. The style guide asks for keyword-only here (.github/instructions/style-guide.instructions.md).

Given #2663 resolves the open issue, I'd suggest closing this one. If you'd like the fairness_bias back-compat alias kept, that's a clean one-line PR on its own after #2663 lands, and I'm happy to review it.

[varunj-msft]

Closing this as superseded — #2663 landed the same ground and covers all three of the #2661 findings.

Thanks again for the contribution, and sorry it didn't end up merging. The fairness_bias back-compat alias is still a reasonable idea on its own if you'd like to open that as a small standalone PR — happy to review it.