Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja by fanny-riols · Pull Request #251 · ServiceNow/eva

fanny-riols · 2026-10-29T19:41:17Z

Why

Native speakers reviewed EVA conversations in 9 languages. Some judge errors they found come from not knowing how the language works.

#249 added a place in each judge prompt where language-specific text can be inserted. #250 changed the rules shared by all languages. This PR writes the language-specific text: one YAML file per judge in configs/prompts/judge_language_guidance/, with one entry per language. When a conversation is in that language, its entry is added to the judge prompt under a ### <Language>-specific guidance heading. English conversations get nothing, so English prompts don't change, except for one line in faithfulness (see below). Canadian French has no entry of its own: it gets the French entry, which includes the Québec items.

How each entry is written. Each entry groups its items by what they mean for the rating, and each group heading says the rating in that judge's own scale. For example, in tts_fidelity: "Correct, does not lower the rating", "Rating = 0 with entity_error".

Version bump

The versions of all metrics whose prompt or source changes in the stack (#249, #250 and this PR) are bumped once here. metrics_version is not changed. tests/fixtures/metric_signatures.json is regenerated.

What changes, per judge

faithfulness (de, nl, fr, es, it, hi, ko, ja)

Letters and digits that sound alike, for each language. Examples: B/D/P/T in German, the 70–99 compound numbers in French ("soixante-dix-sept" = 77, but "soixante, dix-sept" = 60 17), B and V in Spanish, Q (キュー) and 9 (きゅう) in Japanese. The judge uses this list when it decides whether the assistant should have confirmed a value, or passed a wrong value to a tool.
Words borrowed from English that are normal in the language, so the judge doesn't count them as speaking the wrong language. Examples: "Ticket", "Login" in German; Hangul loanwords such as 로그인 in Korean; katakana loanwords such as パスワード in Japanese.
Hindi: English words and short phrases inside a Hindi sentence are normal and never a language violation. Only a whole sentence or turn in English is.
Japanese: an English term in Latin letters where a common Japanese term exists ("sick leave" instead of 病気休暇) is a minor issue, rated 2 and never 1.

agent_speech_fidelity, tts_fidelity, user_speech_fidelity (de, nl, fr, es, it, hi, ko, ja). These three files are generated from one source, so they say the same thing, with the rating wording adapted to each judge's scale. For each language they list:

How letters and digits are named in that language (German J is "jott", French W is "double-vé"), and which English names are often heard instead. If an English name sounds like another letter of the language (English E "ee" sounds like German I), the code has changed and the turn is an error.
Which spoken forms are correct, so the judge doesn't flag them: German and Dutch say units before tens (fünfundsiebzig = 75); French says 80 as "quatre-vingts"; Hindi uses ढाई बजे for 2:30; Korean uses native numbers for hours and Sino-Korean numbers for minutes; Japanese has special readings for counters (4時 is よじ).
Unusual but understandable forms, such as a literal clock reading in Hindi or a digit-by-digit year in Japanese. These are noted but are an error only if a native listener could take them for a different value.
Letters and digits that sound alike, such as M/N/L in German or 1/2 (일/이) in Korean. Audio that is ambiguous between the two is still an error, but the judge must not flag one only because its own transcription shows the other.
Sounds the judge can't tell apart from audio, such as B and V in Spanish, or E (이) and 2 (이) in Korean. The judge must not flag these from audio alone; it checks the intended text or the trace.
Words that are not the wrong language: loanwords, product names, codes. Hindi: English words and phrases inside a Hindi sentence are supported.

conciseness (fr, de, nl, hi, ko, ja)

The normal polite register of each language, which makes sentences longer without making them harder to follow: vous in French, Sie in German, honorifics in Hindi, keigo in Japanese, formal endings in Korean. The judge must not count these as filler.
Set service phrases, one per function, that are not filler: "Je vous écoute" in French, "Einen Moment bitte" in German, 잠시만 기다려 주세요 in Korean, かしこまりました in Japanese.
German: numbers and dates written out in words look long but are one piece of information, so they are not excess length.
Dutch is the exception: its service register is direct, so the politeness exception gives almost no extra room for courtesy wording.
What counts as filler (the same formula twice, stacked acknowledgements, a term followed by its synonym) is in the shared prompt from Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250, so the entries don't repeat it.

speakability (hi, ko, ja)

What a gloss looks like (a term followed by its translation, which Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 made a violation): "कर्मचारी आईडी (employee ID)" in Hindi, "비밀번호(PIN)" in Korean, "暗証番号（PIN）" in Japanese.
What is not a violation: an English word used on its own; Hindi and Japanese punctuation such as । and 「」.

transcription_accuracy_key_entities (de, hi, ko, ja)

Spellings that count as the same value: German ü = ue and ß = ss; Hindi and English digit names; Korean 만-based numbers (1만 2천 = 12,000); Japanese katakana letter names and kana readings of kanji names.
Japanese: Q and 9 are different values even though they sound the same, and a different kanji for the same name reading is wrong once the user has said which kanji.

conversation_progression (ja, ko)

When a code or name has a homophone (Japanese Q and 9, or two kanji for the same name; Korean E and 2, or O and 5), and a lookup fails or the user corrects it, the assistant should ask which one was meant. Asking is never counted as redundant.

Validation

Before is #249, which renders the same prompts as main before this stack. After is this PR, which includes #250.

Unit tests.
Rendered prompts: every judge prompt was rendered in all 10 languages, with this PR and with Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250.

Judge validation datasets (docs/metrics/judge_validation_datasets/), production judge models, 3 runs per side, accuracy / macro-F1, mean ± std:

Judge	Records	Before Judge prompts: per-language guidance slot #249	After this PR
faithfulness (`us.anthropic.claude-opus-4-6-v1`)	137	70.07% ± 0.00% / 65.14% ± 0.26%	67.88% ± 0.73% / 62.18% ± 0.79%
conciseness (`gpt-5.2`)	100	92.41% ± 0.84% / 85.40% ± 1.17%	93.25% ± 0.12% / 86.34% ± 0.71%
conversation_progression (`gpt-5.2`)	136	76.47% ± 2.65% / 74.23% ± 2.95%	73.77% ± 1.12% / 70.78% ± 2.03%
transcription_accuracy_key_entities (`gpt-5.2`), entity F1 / matched accuracy	38	83.14% ± 0.62% / 95.56% ± 0.95%	85.86% ± 1.77% / 95.04% ± 0.61%

The English conversation_progression and transcription_accuracy_key_entities prompts are identical before and after for English, so their differences (up to 3.5 points) show the run-to-run variation of these datasets.
Every other change is within that variation: no regression in English.

Re-scoring existing runs: 134 non-English trials (de, nl, fr, fr-CA, es, it, hi, ko, ja), including the conversations the native-speaker reviewers commented on (150 selected). The column "Noise" below is the variation between two runs of Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 (before adding language guidance) on the same trials, with the same prompt. Agreement with the reviewers: their per-turn comments in German, French, Canadian French, Hindi and Japanese, turned into 234 verdicts saying the rating they saw should be lower, higher or the same; that count is in the "reviewer verdicts". We added "Agree before → after" column to count agreement of judge with reviewers.

Judge	Ratings	Mean score before → after	Ratings changed	Noise	Reviewer verdicts	Agree before → after
agent_speech_fidelity	982 turns	0.995 → 0.969	16 (1.6%)	0.7%	40	18% → 25% / 32%*
user_speech_fidelity	952 turns	0.988 → 0.987	17 (1.8%)	1.2%	21	5% → 19% / 19%*
faithfulness	134 conversations	0.463 → 0.425	23 (17%)	14%	26	50% → 58%
conversation_progression	134 conversations	0.672 → 0.690	25 (19%)	22%	10	40% → 40%
conciseness	992 turns	0.793 → 0.803	175 (18%)	18%	66	32% → 30%

* For agent_speech_fidelity and user_speech_fidelity, "after" agreement uses Gemini 3.8 Flash (gemini-3.8-flash, Use Gemini 3.8 Flash as the default audio judge #263) as the audio judge.

Above noise: agent_speech_fidelity and user_speech_fidelity.
- agent_speech_fidelity is stricter on spelled codes: English letter names in Italian codes ('el', 'dee', 'ef') and V said as 'doppia vu' are flagged, and real errors are caught (an extra digit in a German ID, 'REQ' said as 'EQ' in Dutch).
- user_speech_fidelity catches entity errors such as शून्य dropped from a Hindi ID and an extra 'tre' in an Italian licence number; tts_fidelity flags English digit names in a French code.
speakability flags glosses (violation 5 from Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250), mostly in Japanese: "PIN（暗証番号）", or a code followed by its reading, "INC（アイ・エヌ・シー）". It also flags a code followed by its spelled-out letters in parentheses, "CASE-ACCT-… (C A S E …)", a borderline case that flips between runs.
Within noise: faithfulness, conversation_progression and conciseness. Conciseness changes as often in Spanish and Italian, which have no conciseness guidance.

Stack

Judge prompts: per-language guidance slot #249 Place for language-specific text in judge prompts, and its loader.
Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 Shared judge rules (base of this PR).
Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251 This PR: the language-specific text.
Use Gemini 3.8 Flash as the default audio judge #263 Changes the audio judge from Gemini 3 Flash to Gemini 3.8 Flash.

Adds German guidance to the judge prompts. It is rendered only for German conversations, so other languages are unchanged. - agent_speech_fidelity (de): 10 rules - conciseness (de): 2 rules - faithfulness (de): 5 rules - transcription_accuracy_key_entities (de): 2 rules - tts_fidelity (de): 10 rules - user_speech_fidelity (de): 10 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Dutch guidance to the judge prompts. It is rendered only for Dutch conversations, so other languages are unchanged. - agent_speech_fidelity (nl): 9 rules - conciseness (nl): 2 rules - faithfulness (nl): 4 rules - tts_fidelity (nl): 9 rules - user_speech_fidelity (nl): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds French and Canadian French guidance to the judge prompts. It is rendered only for French and Canadian French conversations, so other languages are unchanged. - agent_speech_fidelity (fr): 9 rules - agent_speech_fidelity (fr-CA): 3 rules - conciseness (fr): 3 rules - conciseness (fr-CA): 1 rule - faithfulness (fr): 3 rules - faithfulness (fr-CA): 2 rules - tts_fidelity (fr): 9 rules - tts_fidelity (fr-CA): 3 rules - user_speech_fidelity (fr): 9 rules - user_speech_fidelity (fr-CA): 3 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Spanish guidance to the judge prompts. It is rendered only for Spanish conversations, so other languages are unchanged. - agent_speech_fidelity (es): 9 rules - faithfulness (es): 4 rules - tts_fidelity (es): 9 rules - user_speech_fidelity (es): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Italian guidance to the judge prompts. It is rendered only for Italian conversations, so other languages are unchanged. - agent_speech_fidelity (it): 8 rules - faithfulness (it): 3 rules - tts_fidelity (it): 8 rules - user_speech_fidelity (it): 8 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Hindi guidance to the judge prompts. It is rendered only for Hindi conversations, so other languages are unchanged. - agent_speech_fidelity (hi): 9 rules - conciseness (hi): 3 rules - faithfulness (hi): 5 rules - speakability (hi): 3 rules - transcription_accuracy_key_entities (hi): 4 rules - tts_fidelity (hi): 9 rules - user_speech_fidelity (hi): 9 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Korean guidance to the judge prompts. It is rendered only for Korean conversations, so other languages are unchanged. - agent_speech_fidelity (ko): 7 rules - conciseness (ko): 2 rules - faithfulness (ko): 4 rules - transcription_accuracy_key_entities (ko): 2 rules - tts_fidelity (ko): 7 rules - user_speech_fidelity (ko): 7 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

Adds Japanese guidance to the judge prompts. It is rendered only for Japanese conversations, so other languages are unchanged. - agent_speech_fidelity (ja): 10 rules - conciseness (ja): 3 rules - conversation_progression (ja): 4 rules - faithfulness (ja): 5 rules - speakability (ja): 3 rules - transcription_accuracy_key_entities (ja): 3 rules - tts_fidelity (ja): 10 rules - user_speech_fidelity (ja): 10 rules Review by a native speaker: check that each rule is correct and natural for the language, and that nothing important is missing.

The guidance slot, the shared rules and the per-language guidance change the prompt or source of these metrics, so their scores are not comparable with earlier versions: - conciseness v0.1 -> v0.2 - conversation_progression v0.2 -> v0.3 - faithfulness v0.3 -> v0.4 - speakability v0.3 -> v0.4 - agent_speech_fidelity v0.4 -> v0.5 - tts_fidelity v0.3 -> v0.4 - user_speech_fidelity v0.2 -> v0.3 - transcription_accuracy_key_entities v0.4 -> v0.5 - user_behavioral_fidelity v0.1 -> v0.2 (source only: shared base class) metrics_version is not changed.

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

- agent_speech_fidelity: drop the entries that rated a clearly spoken but wrong value 0 ('halb drei' for 3:30, soixante-sept for 77, ...). The judge leaves the value to faithfulness and rates articulation only, as in its own "$315" example. The reading facts stay. - tts_fidelity hi and ja: a mispronounced non-entity word is no longer garbled_hallucination, which means unintelligible noise. It rates 0 only if it changes the meaning. - Speech fidelity nl, es, it and ko: the first bullet now lists the English letter and digit names typically heard, as de and fr do, instead of restating the shared rule in three variants. - Example values that occur in the datasets are replaced with synthetic ones (codes, user names, a surname). "M comme Marcel" stays: it is the standard French spelling alphabet. - The metric signature fixture is regenerated.

- Speech fidelity: add the regional 'viertel drei' = 2:15 next to 'drei viertel drei' = 2:45. - Speech fidelity: drop "Meier/Meyer-type spelling variants are fine"; the shared prompt already says it. - Conciseness: keep the written-out date example and drop the restated shared rule.

- Conciseness: "Ik ga meteen kijken" and "Ik ga uw reservering er direct bij zoeken" are single hold phrases, so they are fine on their own. They are filler only when stacked with another hold phrase or lead-in. - Speech fidelity: "bee, cee, dee, pee, tee sound like English" was phonetically wrong (Dutch B is /beː/, close to English "bay"). It now says these written names are the Dutch names. - Speech fidelity: the acronym exemption covers acronyms used as words or names (ID, FMLA, I-9). The letters of a spelled code or identifier still need Dutch names, as the rest of the guidance says. - Speech fidelity: the vowel-change example used 'veer', which is not a digit. The rule now says "makes a digit unrecognisable or turns it into another digit". - Speech fidelity: drop "Upper or lower case need not be spoken", which is not specific to Dutch.

French: - Faithfulness: a code with a compound-number sequence no longer always needs digit-by-digit confirmation. Flag the missing confirmation only if the trace shows the ambiguity mattered, as the shared rule says. - Conciseness: "Je peux vous aider avec ça ?" is a calque; use "Puis-je vous aider ?". Vous-forms are described as normal register instead of restating the shared spoken-length rule. - Speech fidelity: dix-huit is liaison, not elision. 'trois heures et quart' = 3:15 moves here from fr-CA, since it is standard French. Canadian French (adds to the French text, which fr-CA also gets): - Faithfulness: drop the compound-number bullet, which only repeated fr. - Speech fidelity: drop the bullet that repeated the fr and shared rules on English letter names and phrases. - Speech fidelity: the time bullet now gives the Québec forms 'trois heures moins quart' (without 'le') and the colloquial 'quart à trois', both 2:45, so the fr 'moins le quart' entry doesn't reject them. - Conciseness: keep the warm-phrase examples, drop the restated rule.

- Speech fidelity: the B/V bullet said 'be' and 've' sound the same, then called 'be' for V a letter error. Now only a clearly distinguishing name is a letter error: 'uve' / 've corta' said for B, or 'be larga' / 'be de Barcelona' said for V. - Faithfulness: the target is European Spanish, so note that with distinción, C before e/i and Z (/θ/) are distinct from S (/s/). The S/C/Z merge applies to seseo speakers.

Speech fidelity: - "a Italian listener" → "an Italian listener". - The realistic wrong date form is "l'uno marzo", not "il uno marzo". - Spell out the 'Gioia' example: the letter J read as the word 'Gioia' instead of 'i lunga'.

EVA supports Hindi–English code-switching, so the judges no longer penalise English words or phrases inside a Hindi sentence. - Faithfulness: English loanwords, procedural terms ("Employee ID", "last name") and short phrases ("six digits") are never a language violation, in either script. Only a whole English sentence or turn is. This replaces the rule that rated repeated English procedural phrases at most 2, which also contradicted the shared rating scale. The list of Hindi equivalents goes with it, including उपनाम, which usually means nickname rather than last name. - Faithfulness: evidence is a whole English sentence or turn, not Latin script, which on its own says nothing about the language spoken. - Speech fidelity: English words and phrases inside a Hindi sentence are not wrong language. For tts_fidelity and user_speech_fidelity, only a whole English sentence that is not in the intended text is. - Speech fidelity: the mispronunciation examples now say what the wrong pronunciation sounds like ('aisle' said as 'es-le'); the others gave none and are dropped.

Faithfulness: 공 (0) does not sound like the letter O. The real collisions are that the letter O (오) is also the digit 5 (오), and that O is easily confused with 0 in writing.

Speech fidelity: - Counter readings: ななじ is as standard as しちじ for 7時, and じっぷん is a standard reading of 10分; both are added. - く (9) is not confusable with シー (C) or いち (1); drop it from that list. - Keep the katakana examples of a letter group read as a word, and drop the "codes must be read letter by letter" rule, which is not specific to Japanese.

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

…-language-guidance-content

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

The fabricating_tool_parameters note on confusable letters now points to the language-specific guidance, which this branch adds. The note is in the shared prompt, so the English faithfulness prompt gets the pointer too; "if any" covers languages without guidance.

Group each language entry by what its items mean for the rating, with the rating in the judge's own scale in each group heading. Move the fr-CA entries into fr, so Canadian French gets the French guidance only. Carry confusable pairs and homophones already listed for a language in one judge over to that language's entries in the other judges, and add Korean entries for speakability glosses and conversation_progression homophones.

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

…es agree Dutch: an acronym used as a word in a sentence may use English letter names, but the letters of a spelled code still follow the English letter names rule. Hindi: English digit names are correct when the text writes the digit as a numeral or in English, and only unusual when it spells out the Hindi digit word. agent_speech_fidelity, tts_fidelity and user_speech_fidelity share their per-language facts; a test now checks that the groups whose items do not depend on the rating scale list the same items in all three files.

Items such as "Stacked acknowledgements: Parfait, très bien, d'accord, merci beaucoup" or a list of loanwords could be read as the only cases to flag or to accept. They now say they are examples (e.g., such as, for example), and the loanword headings say the list is not complete.

…rd note for ko and hi Merge the latest shared judge rules. Dutch conciseness no longer lists lead-ins that announce the next step as filler. Korean and Hindi faithfulness now say, as Japanese does, that a transcript rendering a spoken loanword in Latin letters is not evidence of English speech.

tiffanyannedo

Reviewed JA additions to the prompts. Made some comments/suggestions.

tiffanyannedo · 2026-11-12T17:19:40Z

+  **Correct, does not lower the rating:**
+  - These are the Japanese names of Latin letters in a spelled code, in katakana; a letter said this way is correct, even with an English-like accent: エー, ビー, シー, ディー, イー, エフ, ジー, エイチ/エッチ, アイ, ジェー, ケー, エル, エム, エヌ, オー, ピー, キュー, アール, エス, ティー, ユー, ブイ/ヴィー, ダブリュー, エックス, ワイ, ゼット/ズィー.
+  - These are the Japanese digit names in a spelled code; a digit said this way is correct: ゼロ/れい, いち, に, さん, よん, ご, ろく, なな, はち, きゅう.
+  - This is how numbers are read with counters in Japanese; a number read this way is correct (し, しち, く are right here): 4時 よじ, 7時 しちじ/ななじ, 9時 くじ, 4月 しがつ, 7月 しちがつ, 9月 くがつ, 1日 ついたち, 2日 ふつか, 4日 よっか, 8日 ようか, 20日 はつか, 24日 にじゅうよっか, 1分 いっぷん, 10分 じゅっぷん/じっぷん, 2人 ふたり, 4人 よにん.


ななじ is not a way to say ７時. But, 7月 can be said as なながつ.
Dates all have special readings. Should this be exhaustive?
じっぷん is not a way to say 10分 and mean the time/duration, it is a way to say "enough"
Same with counting people, should be this be exhaustive?

Suggested change

- This is how numbers are read with counters in Japanese; a number read this way is correct (し, しち, く are right here): 4時よじ, 7時しちじ/ななじ, 9時くじ, 4月しがつ, 7月しちがつ, 9月くがつ, 1日ついたち, 2日ふつか, 4日よっか, 8日ようか, 20日はつか, 24日にじゅうよっか, 1分いっぷん, 10分じゅっぷん/じっぷん, 2人ふたり, 4人よにん.

- This is how numbers are read with counters in Japanese; a number read this way is correct (し, しち, く are right here): 4時よじ, 7時しちじ, 9時くじ, 4月しがつ, 7月しちがつ/なながつ, 9月くがつ, 1日ついたち, 2日ふつか, 3日みっか, 4日よっか, 5日いつか, 6日むいか, 7日なのか, 8日ようか, 9日ここのか, 10日とおか, 14日じゅうよっか, 20日はつか, 24日にじゅうよっか, 1分いっぷん, 10分じゅっぷん, 1人ひとり, 2人ふたり, 4人よにん.

How exhaustive would that be? If it's just as big as these rows, then yes, please.

Yes, the suggestions are all the ones I think we should add

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

Co-authored-by: tiffanyannedo <142948481+tiffanyannedo@users.noreply.github.com>

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

… informative English clause is rated 1

… your own transcript

…o, not by your own transcript

…, not by your own transcript

…ntent' into pr/fr/judge-language-guidance-content # Conflicts: # configs/prompts/judge_language_guidance/agent_speech_fidelity.yaml # configs/prompts/judge_language_guidance/tts_fidelity.yaml # configs/prompts/judge_language_guidance/user_speech_fidelity.yaml # tests/fixtures/metric_signatures.json

…the Japanese review, as in tts_fidelity and user_speech_fidelity

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

… with kanji; invalid name forms (review suggestions)

…t outside IT (Spanish review)

…de times, same-sound pairs in names (Spanish review)

…añana/tarde times, same-sound pairs in names (Spanish review)

…ñana/tarde times, same-sound pairs in names (Spanish review)

…ith them; drop the Québec register line

fanny-riols added 9 commits October 29, 2026 15:38

This was referenced Oct 29, 2026

Judge prompts: per-language guidance slot #249

Open

Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250

Open

fanny-riols added 19 commits October 29, 2026 16:38

Merge branch 'pr/fr/judge-shared-multilingual-rules' into pr/fr/judge…

469683c

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

Italian judge guidance: review fixes

0c4d49d

Speech fidelity: - "a Italian listener" → "an Italian listener". - The realistic wrong date form is "l'uno marzo", not "il uno marzo". - Spell out the 'Gioia' example: the letter J read as the word 'Gioia' instead of 'i lunga'.

Korean judge guidance: review fix

29b5767

Faithfulness: 공 (0) does not sound like the letter O. The real collisions are that the letter O (오) is also the digit 5 (오), and that O is easily confused with 0 in writing.

Merge branch 'pr/fr/judge-shared-multilingual-rules' into pr/fr/judge…

ab70b61

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

Merge branch 'pr/fr/judge-shared-multilingual-rules' into pr/fr/judge…

c5ad98a

…-language-guidance-content

Merge branch 'pr/fr/judge-shared-multilingual-rules' into pr/fr/judge…

8e60a41

…-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

6c80c14

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

fanny-riols mentioned this pull request Nov 12, 2026

Use Gemini 3.8 Flash as the default audio judge #263

Open

2 tasks

tiffanyannedo requested changes Nov 12, 2026

View reviewed changes

fanny-riols and others added 5 commits November 12, 2026 13:37

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

5f28add

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

38cd82a

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/metric_signatures.json

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

4e5253f

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

Apply batched suggestions from code review for Japanese

d67cb27

Co-authored-by: tiffanyannedo <142948481+tiffanyannedo@users.noreply.github.com>

Apply pre-commit

135cb51

fanny-riols requested a review from tiffanyannedo November 12, 2026 20:44

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge_language_guidance/faithfulness.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge_language_guidance/faithfulness.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge_language_guidance/faithfulness.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread src/eva/metrics/accuracy/faithfulness.py Resolved

lmendezsn-cmyk reviewed Nov 12, 2026

View reviewed changes

fanny-riols added 8 commits November 13, 2026 10:55

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

4dc7205

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

Faithfulness guidance: label language items wrong_language; Hindi: an…

82a75bd

… informative English clause is rated 1

tts_fidelity guidance: judge sound-alike letters by the audio, not by…

1c66d18

… your own transcript

agent_speech_fidelity guidance: judge sound-alike letters by the audi…

4a08b49

…o, not by your own transcript

user_speech_fidelity guidance: judge sound-alike letters by the audio…

0846964

…, not by your own transcript

agent_speech_fidelity guidance (ja): apply the counter readings from …

218b72d

…the Japanese review, as in tts_fidelity and user_speech_fidelity

Merge remote-tracking branch 'origin/pr/fr/judge-shared-multilingual-…

69b3cb5

…rules' into pr/fr/judge-language-guidance-content # Conflicts: # tests/fixtures/judge_prompts_en.json # tests/fixtures/metric_signatures.json

Copilot started work on behalf of fanny-riols November 13, 2026 16:26

View session

Copilot finished work on behalf of fanny-riols November 13, 2026 16:27

fanny-riols added 7 commits November 13, 2026 11:30

transcription_accuracy_key_entities guidance (ja): all digit readings…

63ff2f3

… with kanji; invalid name forms (review suggestions)

Faithfulness guidance (es): Z in all positions /θ/, C/K/QU /k/, ticke…

8c12018

…t outside IT (Spanish review)

tts_fidelity guidance (es): i latina, doble uve, ve chica, mañana/tar…

8a135b0

…de times, same-sound pairs in names (Spanish review)

agent_speech_fidelity guidance (es): i latina, doble uve, ve chica, m…

66ca881

…añana/tarde times, same-sound pairs in names (Spanish review)

user_speech_fidelity guidance (es): i latina, doble uve, ve chica, ma…

044318b

…ñana/tarde times, same-sound pairs in names (Spanish review)

Faithfulness guidance (es): ticket listed without context

e38cdfe

Faithfulness guidance: sound-alike headings say what the judge does w…

9e5a2cd

…ith them; drop the Québec register line

Conversation

fanny-riols commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Version bump

What changes, per judge

Validation

Stack

Uh oh!

tiffanyannedo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tiffanyannedo Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

fanny-riols Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

tiffanyannedo Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fanny-riols commented Oct 29, 2026 •

edited

Loading