Add metrics dashboard: scorer quality and benchmark leaderboard - #2667
Draft
hannahwestra25 wants to merge 6 commits into
Draft
hannahwestra25 wants to merge 6 commits into
hannahwestra25 wants to merge 6 commits into
Conversation
…fields Adds a new doc/dashboard/ section with a Scorer Quality page (Objective Scorer Leaderboard + Harm Scorer Leaderboard), rendered from the existing committed pyrit/datasets/scorer_evals/ registries. Extends build_scripts/export_adversarial_benchmark_result.py to attach objective_target/objective_scorer/dataset identity fields to every technique-metrics row, and adds an optional --update-benchmark-store flag that upserts rows into a new committed JSONL store (pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl), keyed on (technique, adversarial_model, objective_target, objective_scorer, dataset). Scope, intentionally: - No ADO pipeline/YAML changes. - No benchmark leaderboard dashboard page yet - the exporter change is groundwork for a future PR once that page and real exported data exist. --update-benchmark-store has been unit-tested with hand-constructed ScenarioResult/AttackResult/identifier objects (verified field-for-field against source) but not yet exercised against a real AdversarialBenchmark scenario run, since that requires live credentialed targets and has no consumer yet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
hannahwestra25
marked this pull request as draft
September 15, 2026 18:46
- doc/dashboard/2_benchmark_leaderboard.py/.ipynb: new page rendering a technique / adversarial-model leaderboard (success rate, N, and success/failure/error/undetermined counts) from pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl. - pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl: new store populated with 3 rows from one real, live AdversarialBenchmark scenario run (role_play_video_game, context_compliance, red_teaming; one adversarial target; harmbench; --max-dataset-size 1) via export_adversarial_benchmark_result.py --update-benchmark-store. This is genuine exporter output, not synthetic/mocked data. - doc/myst.yml: register the new page under the dashboard TOC. - doc/dashboard/0_dashboard.md: move Benchmark Leaderboard from "planned" to "here today", document its refresh procedure, and note the future objective-target robustness leaderboard as the next step. Demo-scale caveat: N=1 per technique and a single adversarial-model identity (mirroring the objective target's Azure deployment, since no independent ADVERSARIAL_CHAT_* target was available). This validates the exporter/store/page mechanism end-to-end for real, but is not yet a statistically meaningful or cross-model benchmark. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Both the Scorer Quality and Benchmark Leaderboard notebooks now render their tables via a shared render_leaderboard_card() helper (dark GitHub-style card CSS, static HTML/no JS) instead of plain print(df.to_string()), matching the dashboard mockups. Also fixes two bugs found while wiring this up: - 2_benchmark_leaderboard.py: the render call was nested inside an if/else block, so IPython's last-expression auto-display never fired and the card silently rendered nothing. Now wrapped in an explicit display(...) call. - _format_cell in both notebooks treated percent columns as percentages only when the underlying value was a Python float, but pandas infers int64 for an all-whole-number JSON column (e.g. success_rate values of 0.0/1.0), so percents rendered as literal "1"/"0". percent_columns membership is now authoritative regardless of the pandas-inferred dtype. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Scorer Quality and Benchmark Leaderboard pages were .ipynb/.py
jupytext pairs whose rendered output showed a large wall of raw Python
source (imports, CSS strings, helper functions) above each leaderboard
table -- readers had to scroll past implementation details to see the
actual metrics.
Replace both with plain MyST Markdown pages that {include} pre-rendered
HTML fragments instead:
- build_scripts/generate_dashboard_html.py: new script that renders the
same three leaderboard cards (Objective Scorer, Harm Scorer, Benchmark)
the notebooks used to produce, and writes each as a standalone HTML
fragment under doc/dashboard/_generated/. Faithfully ports the existing
rendering logic (dark card CSS, percent-formatting fix, empty-state
fallback for when the benchmark store doesn't exist yet).
- doc/dashboard/1_scorer_quality.md, 2_benchmark_leaderboard.md: new
pages carrying the same prose as the old notebooks, with
`{include}` directives in place of code cells.
- doc/dashboard/_generated/*.html: the generated fragments, committed
like any other generated-and-checked-in artifact (comparable to the
notebooks' own committed outputs today).
- doc/dashboard/0_dashboard.md: updated links (.ipynb -> .md) and
rewrote "Refreshing the data" to describe running
python -m build_scripts.generate_dashboard_html instead of
re-executing a notebook.
- doc/myst.yml: point the TOC at the new .md files.
- Deleted the old 1_scorer_quality.py/.ipynb and
2_benchmark_leaderboard.py/.ipynb pairs.
CI never executes notebooks today (jupyter-book build --all --html,
no --execute flag), and .github/instructions/docs.instructions.md
only applies to .py/.ipynb pairs, so this needed no CI changes --
pre-generate once, commit the static HTML, and the site build just
serves it, exactly like it already does for notebook outputs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…n script - resolve_objective_identity() in adversarial.py derives stable (objective_target, objective_scorer) display labels, shared by the scenario's own result-reuse cache and the exporter. - AdversarialBenchmark can now recognize combinations already present in a committed benchmark_store_path and skip re-running them. - export_adversarial_benchmark_result.py gains --update-benchmark-store to upsert technique-metrics rows (keyed on technique/adversarial_model/ objective_target/objective_scorer/dataset) into the committed JSONL store. - New build_scripts/run_adversarial_benchmark.py: a re-runnable wrapper that skips combinations already cached (in the store or live memory), so adding a new adversarial target only executes the new combinations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- New build_scripts/import_adversarial_benchmark_snapshot.py upserts an externally-produced technique-metrics.json (e.g. downloaded from an Azure DevOps pipeline artifact) into the committed benchmark metrics store, reusing the same upsert_benchmark_metrics() as the local exporter. - Supports --default-objective-target/--default-objective-scorer/ --default-dataset to backfill identity fields on rows produced by an exporter version that predates them, without overwriting a field a row already has. - Ingested the 6 real technique-metrics rows from Azure DevOps build 16150 (a preview run of PR microsoft#2551's not-yet-merged dataset/pipeline updates), backfilling objective_target=openai_chat and dataset=adversarial_benchmark_v1 per the run's actual pipeline parameters; objective_scorer is <unknown> since it wasn't captured by that pipeline version. - Documented the provenance of these rows and the new ingestion path in the dashboard docs, and regenerated the benchmark leaderboard HTML fragment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Metrics dashboard (
doc/dashboard/) — a new top-level docs section:0_dashboard.md: landing page explaining what's here, what's planned, and how to refresh the data.1_scorer_quality.py/.ipynb: an Objective Scorer Leaderboard (accuracy/F1/precision/recall) and a Harm Scorer Leaderboard (MAE, Krippendorff's alpha), rendered from the existing committed registries underpyrit/datasets/scorer_evals/. No new dependencies — reads local JSONL only.2_benchmark_leaderboard.py/.ipynb: a Technique / Adversarial Model Leaderboard (success rate, N, success/failure/error/undetermined counts), rendered from the new committed store below.doc/myst.yml's top-level TOC, afterscanner.Benchmark exporter identity fields + store (
build_scripts/export_adversarial_benchmark_result.py):objective_target,objective_scorer, anddatasetidentity fields (added to JSON/CSV/TXT output).--update-benchmark-store [--benchmark-store-path]flag that upserts rows into a new committed store,pyrit/datasets/benchmark_results/adversarial_benchmark_metrics.jsonl, keyed on(technique, adversarial_model, objective_target, objective_scorer, dataset). This is what the new Benchmark Leaderboard page reads.Demo data behind the Benchmark Leaderboard: three rows from one real, live
AdversarialBenchmarkscenario run —role_play_video_game,context_compliance, andred_teamingtechniques, one adversarial target,harmbenchdataset,--max-dataset-size 1. This is genuine exporter/scenario output, not synthetic data (see caveat below).Explicitly out of scope for this PR
0_dashboard.md). The existing manually-triggered.azuredevops/adversarial-benchmark.ymlpipeline could in principle drive this, but today its export step never passes--update-benchmark-store, and it has no step to regenerate notebook outputs or to commit/push/open a PR (it has no GitHub write credentials). Wiring that up is a good follow-up, best done as its own change rather than bundled here — it also touches the same YAML file Improve adversarial benchmark dataset and scoring #2551 is actively modifying.Known limitations
N=1per row) using one adversarial-model identity that happens to share a deployment with the objective target (no independently-configuredADVERSARIAL_CHAT_*target was available in this environment). Treat the page as a working demonstration of the mechanism, not a statistically meaningful or cross-model comparison. Broadening this into a larger, regularly-refreshed sweep across real independent adversarial models is future work (see the page's "Note on scope" section)._objective_identity,_dataset_identity,_build_technique_metrics,_upsert_benchmark_metrics) is covered by unit tests using hand-constructedScenarioResult/AttackResult/TargetIdentifier/ScorerIdentifierobjects, and has now also been exercised end-to-end against the real scenario result described above — so both the isolated logic and the full memory-to-JSONL path are validated against genuine PyRIT output.Testing
pytest tests/unit/build_scripts/test_export_adversarial_benchmark_result.py— 12 new tests, all passing.pytest tests/unit/build_scripts tests/unit/common tests/unit/datasets— no regressions (268 + 5069 passed, 2 pre-existing skips).ruff check/ruff format --check— clean.python -m build_scripts.validate_docs— 163 TOC references validated, no orphaned files.pre-commit runagainst all changed files — clean, aside from two confirmed pre-existing, environment-specific issues unrelated to this change:check-added-large-filesfails against any file (including already-committed ones) due to agit check-attrsubprocess error in this sandbox;nbstripoutreports "modified" on every run even though a before/after hash comparison shows the file is byte-identical.AdversarialBenchmarkscenario end-to-end (pyrit_scan run→export_adversarial_benchmark_result.py --update-benchmark-store→ the new dashboard page) to validate the full pipeline against genuine data rather than mocks.