[feat]: Adding Meta-Eval pipeline into the framework

[kargibora]

Problem

JudgeArena can use judges to evaluate models, but it needs a way to evaluate the judges themselves against human preferences. This PR adds that comparison within the existing task and metric framework.

What this PR introduces

Meta-evaluation tasks for LMArena-100k, LMArena-140k, and ComparIA, with three types of comparison:

• Agreement: accuracy and Cohen’s kappa against human votes, with parse coverage reported separately.
• Ranking: Spearman correlation and Elo MAE between human and judge rankings.
• Elo-gap: held-out Elo error at different annotation budgets, using hard and soft preferences.

All of these metrics can be improved,removed or added easily - for example one can easily add Kendells-$\tau$ to the correlation without changing the pipeline.

How it works

The runner selects the most frequent models and samples a connected set of battles with a per-model quota. It judges those battles through the existing prompt and parsing pipeline, then passes canonical numeric preferences to the configured metrics.

The implementation reuses the existing dataset loaders, ParsedPreference, Bradley–Terry fitter, and metric/report APIs. Task YAML defines the dataset, prompt, and metric settings. Runtime options control sampling.

Both fixed and two-orientation judging are supported. With two orientations, both passes must parse before a battle contributes a preference. Failed judgments count against attempted agreement accuracy rather than disappearing from the denominator.
For example:

uv run judgearena \
  --task meta-eval-lmarena-140k-en \
  --judge.model OpenRouter/deepseek/deepseek-v3.2 \
  --judge.temperature 0 \
  --judge.swap_mode fixed \
  --meta_eval.top_models 10 \
  --meta_eval.battles_per_model 50 \
  --run.seed 0 \
  --run.result_folder results/meta-eval

The run saves the sampled battles, judge annotations, metric inputs, and results for inspection.

Size and validation

Relative to #115 at e38ccae. Parent changes are excluded. Pre-existing tests are restored; remaining test reductions apply only to stack-added coverage.

File type	Added	Deleted	Changed
Production Python	1,486	77	1,563
Tests	601	6	607
Task YAML, prompts, README	199	1	200
Total	2,286	84	2,370

Most core additions are in scoring (576 lines), the runner (217), annotation (184), and sampling (164).

Full local suite: 389 passed.

[kargibora]

This is now re-used with the elo. No need for duplication