Skip to content

Use Gemini 3.8 Flash as the default audio judge #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fanny-riols wants to merge 11 commits into
base: pr/fr/judge-language-guidance-content
Choose a base branch
from
+15 −15
Open

Use Gemini 3.8 Flash as the default audio judge #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
708d485
Use Gemini 3.8 Flash as the default audio judge
fanny-riols Nov 12, 2026
089c949
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 12, 2026
ae38965
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 12, 2026
edb9213
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 12, 2026
e626c7b
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
deb20f1
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
a2e233c
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
6802c72
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
56fa9a8
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
6d9f927
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
77ba33b
Merge remote-tracking branch 'origin/pr/fr/judge-language-guidance-co…
fanny-riols Nov 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/experiment_setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ The table below lists the API-hosted models used as a judge.
| Model ID | Provider | Type | Parameters |
|---------------------------------------------|-------------|------|-----------------------------|
| gpt-5.2 | OpenAI | LLM | reasoning: default |
| gemini-3-flash-preview | Google | LLM | reasoning: default |
| gemini-3.8-flash | Google | LLM | reasoning: default |
| us.anthropic.claude-opus-4-6-v1 | AWS Bedrock | LLM | reasoning: default |


Expand Down
2 changes: 1 addition & 1 deletion docs/metrics/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ Each metric implements `BaseMetric.compute(context: MetricContext) -> MetricScor
**LLM-as-Judge:**
- Integer/boolean ratings (not decimals) to avoid precision issues
- Structured prompts in `configs/prompts/judge.yaml`
- GPT-5.2 for text judges, Gemini 3 Flash for audio judges, Claude Opus for faithfulness
- GPT-5.2 for text judges, Gemini 3.8 Flash for audio judges, Claude Opus for faithfulness

**Audio Evaluation:**
- Audio encoded as base64 WAV, sent to Gemini via LiteLLM
Expand Down
4 changes: 2 additions & 2 deletions docs/metrics/agent_speech_fidelity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To keep the EVA score **apples-to-apples across all pipeline setups**, the same
### Evaluation Method

- **Type**: Audio Judge (multimodal LLM with audio input)
- **Model**: Gemini 3 Flash
- **Model**: Gemini 3.8 Flash
- **Granularity**: Per-turn (each assistant turn evaluated independently)

### Input Data
Expand Down Expand Up @@ -71,4 +71,4 @@ The judge receives the agent audio plus a redacted conversation trace and, for e
- **Class**: `SpeechFidelityMetric`
- **Base Class**: `SpeechFidelityBaseMetric` → `AudioJudgeMetric`
- **Prompt**: `configs/prompts/judge.yaml` under `judge.agent_speech_fidelity`
- **Configuration**: `audio_judge_model` (default: Gemini 3 Flash), `aggregation` (default: "mean")
- **Configuration**: `audio_judge_model` (default: Gemini 3.8 Flash), `aggregation` (default: "mean")
4 changes: 2 additions & 2 deletions docs/metrics/tts_fidelity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Audio-based metric that evaluates whether the assistant's **spoken audio** accur
### Evaluation Method

- **Type**: Audio Judge (multimodal LLM with audio input)
- **Model**: Gemini 3 Flash
- **Model**: Gemini 3.8 Flash
- **Granularity**: Per-turn (each assistant turn evaluated independently)

### Input Data
Expand Down Expand Up @@ -79,4 +79,4 @@ The judge compares intended text against spoken audio, focusing on:
- **Class**: `TTSFidelityMetric`
- **Base Class**: `SpeechFidelityBaseMetric` → `AudioJudgeMetric`
- **Prompt**: `configs/prompts/judge.yaml` under `judge.tts_fidelity`
- **Configuration**: `audio_judge_model` (default: Gemini 3 Flash), `aggregation` (default: "mean")
- **Configuration**: `audio_judge_model` (default: Gemini 3.8 Flash), `aggregation` (default: "mean")
4 changes: 2 additions & 2 deletions docs/metrics/user_speech_fidelity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Audio-based validation metric that evaluates whether the user simulator's **spok
### Evaluation Method

- **Type**: Audio Judge (multimodal LLM with audio input)
- **Model**: Gemini 3 Flash
- **Model**: Gemini 3.8 Flash
- **Granularity**: Per-turn (each user turn evaluated independently)

### Input Data
Expand Down Expand Up @@ -67,5 +67,5 @@ This metric uses a 1-3 scale instead of binary 0-1 (like agent speech fidelity)
- **Prompt location**: `configs/prompts/judge.yaml` under `judge.user_speech_fidelity`
- Uses the same speech fidelity prompt structure as `agent_speech_fidelity` but with `evaluation_mode="user"` and user turns
- **Configuration options**:
- `audio_judge_model`: LLM model (default: Gemini 3 Flash)
- `audio_judge_model`: LLM model (default: Gemini 3.8 Flash)
- `aggregation`: Aggregation method (default: "mean")
2 changes: 1 addition & 1 deletion src/eva/metrics/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -662,7 +662,7 @@ class AudioJudgeMetric(BaseMetric):
metric_type = MetricType.AUDIO_JUDGE

# Subclasses can override
default_model = "gemini-3-flash-preview"
default_model = "gemini-3.8-flash"
default_params: dict[str, Any] = {"temperature": 0.0, "max_tokens": 40000, "reasoning_effort": "minimal"}
rating_scale: tuple[int, int] = (-2, 2) # Can vary by metric

Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/metric_signatures.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,13 @@
"SpeechFidelityMetric": {
"name": "agent_speech_fidelity",
"prompt_hash": "3668f89a94f8",
"source_hash": "a3bfd5b63aba",
"source_hash": "c4e6f3d0b3cf",
"version": "v0.5"
},
"TTSFidelityMetric": {
"name": "tts_fidelity",
"prompt_hash": "b8a5fc23cbb2",
"source_hash": "bf688bc70e3a",
"source_hash": "4846ca933c54",
"version": "v0.4"
},
"TaskCompletion": {
Expand Down Expand Up @@ -98,7 +98,7 @@
"UserSpeechFidelityMetric": {
"name": "user_speech_fidelity",
"prompt_hash": "df4cf200e659",
"source_hash": "97456368615c",
"source_hash": "ec8ad53652e0",
"version": "v0.3"
}
}
6 changes: 3 additions & 3 deletions website/src/data/metricsData.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ export const metrics: MetricDefinition[] = [
badge: 'beta',
category: 'eva-a',
type: 'lalm_judge',
judgeModel: 'Gemini 3 Flash',
judgeModel: 'Gemini 3.8 Flash',
judgeAccuracy: 0.8957,

@katstankiewicz katstankiewicz Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do the accuracy labels need updating?

@fanny-riols fanny-riols Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it's one of the PR description todos :)

judgeScores: [
{ label: 'accuracy', value: 0.8957, std: 0.0258 },
Expand Down Expand Up @@ -862,7 +862,7 @@ Transcribed: \`My phone number is 404-555.\`
displayName: 'TTS Fidelity',
category: 'debug',
type: 'lalm_judge',
judgeModel: 'Gemini 3 Flash',
judgeModel: 'Gemini 3.8 Flash',
judgeAccuracy: 0.8957,
judgeScores: [
{ label: 'accuracy', value: 0.8957, std: 0.0258 },
Expand Down Expand Up @@ -1119,7 +1119,7 @@ Respond in JSON format:
displayName: 'User Speech Fidelity',
category: 'validation',
type: 'lalm_judge',
judgeModel: 'Gemini 3 Flash',
judgeModel: 'Gemini 3.8 Flash',
description: 'Audio-based check that the user simulator\'s TTS output matches the intended text. Validates that the simulated user is actually saying what it\'s supposed to say.',
inputs: 'User audio recording, intended user text',
outputRange: '1-3 per turn (1=low fidelity, 2=medium, 3=high), aggregated as mean',
Expand Down
Loading