Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness by fanny-riols · Pull Request #250 · ServiceNow/eva

fanny-riols · 2026-10-29T19:41:14Z

Why

Some of the problems the native-speaker reviewers found are not specific to one language. They would happen in any language, including English, so they belong in the shared judge prompts rather than in per-language guidance.

This PR changes English scoring for all six judges it touches: conciseness, faithfulness, speakability, tts_fidelity, agent_speech_fidelity and user_speech_fidelity.

Versions are bumped in #251, so this PR must not be run or merged on its own.

What changes

Judge	Change
conciseness	- The politeness exception also covers formal register and set service phrases: one phrase per function (greet / open, acknowledge, hold, return from hold, thank, close), and a single courtesy phrase is never filler. Repetition is still filler: the same formula twice, stacked acknowledgements, two hold phrases stacked together ("One moment, let me check that for you, just a second"; a single "One moment, let me check that" is one hold phrase), or a term followed by its synonym or its translation into another language. - Expanding an acronym once, on its first mention ("your PTO, that's your paid time off"), is not repetition. - Length is judged by what the listener must process: the number of distinct facts, options, steps and requests, and runaway sentences that chain or stack clauses, not written length, sentence count or word count ("2–4 sentences" is a rough guide). Polite constructions and written-out numbers can look long without adding cognitive load.
faithfulness	- New dimension `wrong_language` (six dimensions instead of five): speech in a language other than the one in the agent instructions, including a phrase or clause inside a correct-language turn ("one moment", "Let me check that" in a French call). Not a flag: established loanwords, codes, IDs, acronyms, product names and proper nouns. It replaces the language bullet of `violating_policies` and adds the `faithfulness.wrong_language_rate` sub-metric. - Severity, in the Rating Scale with the other dimensions: an isolated fixed formulaic phrase (hold, acknowledgement, courtesy), even when it recurs, or a slip that is corrected, is rated 2; whole sentences of content, a clause that carries information the user needs ("Je regarde votre dossier, and I found your booking for Tuesday."), or most turns in another language are rated 1. - `failing_to_disambiguate`: sound-alike letters or digits in a value are not a flag by themselves. When the trace shows the ambiguity mattered (failed lookup, correction, two versions), trying the confusable variants or asking the user to spell is correct; doing neither is the existing "failing to consider possible input errors" flag. - Where the language-specific guidance section (from #251) states an exception, the judge follows it.
speakability	New violation 5, glosses: a term followed, in parentheses or after a slash, by its translation into another language or an exact synonym ("PIN / passcode"), so the listener hears the same thing twice. Not a violation: an acronym expanded once on first mention ("paid time off (PTO)"), a slash between paired terms ("AM/PM", "and/or"), or a word used on its own.
tts_fidelity	Spelled-code and mixed-language rules added to the existing sections (below).
agent_speech_fidelity	Same rules, limited to trace entities, on the 0/1 scale (below).
user_speech_fidelity	Same rules on the 1–3 scale (below).

Spelled codes and mixed-language speech (added to each prompt's error, ignore and wrong-language sections):

Errors:
- A letter or digit in a spelled code or acronym said with its name from another language (an English letter name in a German code) is wrong language. Codes and acronyms are not an exception. If the name sounds like a different letter of the expected language, it is also an entity error.
- A spelled code read as a word instead of letter by letter ("INC" read as "ink"), or a letter in it said as an ordinary word (Italian "J" said as "Gioia" instead of "i lunga"), is an entity error.
- A code normally read digit by digit (PIN, phone, confirmation or account number) read as one number ("one thousand two hundred thirty-four" for PIN 1234), or any reading a native listener could take for a different value, is an entity error.
- Audio that could be either of two sound-alike letters or digits is an error.
Not errors:
- Spelling-alphabet words ("A as in Anna").
- The intended words read in an unusual order or grouping (digit by digit vs as one number), unless it is one of the code readings above.
- Sound-alike letters: the judge goes by what the audio says, not by its own transcript, so clear audio of the intended letter is correct even if it transcribed the other one.
- Proper nouns, official product or system names, established loanwords, and words in another language that are in the intended text.
Per judge:
- tts_fidelity: a phrase or sentence in another language that is not in the intended text is wrong language (rating 0).
- user_speech_fidelity: anything else spoken in another language (a word, a phrase, a letter name) is at most 2, and 1 if it changes an entity or the turn no longer makes sense; the description of rating 2 covers this case.
- agent_speech_fidelity: only the entities must be said in the expected language; speech in another language around them is left to faithfulness and tts_fidelity.
Where the language-specific guidance section (from Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251) states an exception (code-switching or digit names that are normal in that language), the judge follows it.

tests/fixtures/judge_prompts_en.json records the new English renders of the six judges.

Validation

English judge validation datasets (docs/metrics/judge_validation_datasets/), production judge models, 3 runs per side, mean ± std. Before is Judge prompts: per-language guidance slot #249, which renders the same English prompts as main.

Judge	Records		Accuracy	Macro-F1
faithfulness (`us.anthropic.claude-opus-4-6-v1`)	137	before	70.07% ± 0.00%	65.14% ± 0.26%
		after	69.34% ± 1.26%	64.01% ± 1.18%
conciseness (`gpt-5.2`)	100	before	92.41% ± 0.84%	85.40% ± 1.17%
		after	92.76% ± 1.15%	84.48% ± 2.86%

Both changes are within one standard deviation: no English regression on these datasets. The dataset runner has no audio judges, and there is no English dataset for speakability, tts_fidelity or user_speech_fidelity. They are covered by the re-scoring below.

Re-scoring existing runs, before (Judge prompts: per-language guidance slot #249) and after (Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250):

Which runs: the 310 conversations the native-speaker reviewers commented on (de, es, fr, fr-CA, hi, it, ja, ko). For the audio judges, also existing runs in de, fr, fr-CA, hi, ja and English with spelled codes or names with diacritics. For conciseness, also runs with several hold phrases in one turn.
Noise: Judge prompts: per-language guidance slot #249 was scored a second time, so each change can be compared with run-to-run variation.
Speakability (gpt-5.2): glosses are now flagged. 66 of the 70 turns that moved from pass to fail are a term followed by its translation or exact synonym, such as a native term with its English equivalent in parentheses. Why: the new violation 5 names glosses, since the listener hears the same thing twice. Acronyms expanded on first mention and slash pairs like "AM/PM" are excluded, so they keep passing.
Conciseness (gpt-5.2): a small shift toward lenient, from 0.812 to 0.829. 226 turns moved up against 148 down, while a re-run on the same code splits evenly (182 / 180). verbosity_or_filler flags go from 552 to 502 and excess_information_density from 208 to 181. Why: one courtesy phrase per function and one first-mention acronym expansion are allowed, and politeness conventions are said to add length "without adding cognitive load". So turns that are polite but not repetitive are no longer counted as filler, while repeated or stacked formulas still are.
Faithfulness (us.anthropic.claude-opus-4-6-v1):
- Wrong-language phrases are rated by severity. An isolated English phrase in another language, such as "Let's do it" in a German call, now gets 2 instead of 3, and recurring fixed phrases that were rated 1 now get 2. Why: Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 separates a fixed phrase (a hold, acknowledgement or courtesy formula), rated 2, from whole sentences or most turns in another language, rated 1.
- There are fewer confusability flags: conversations flagged for failing_to_disambiguate go from 63 to 53, and for fabricating_tool_parameters from 48 to 40. Why: sound-alike letters or digits are flagged only when the conversation shows the ambiguity mattered.

Speech fidelity (gemini-3-flash-preview): the three audio judges catch more real entity errors and change little otherwise.

Judge	Records with spelled codes or names (turns)	Mean before → after	Turns changed	Noise
agent_speech_fidelity	91 (661)	0.961 → 0.942	8	3
tts_fidelity	64 (503)	0.969 → 0.973	13	5
user_speech_fidelity	88 (613)	0.985 → 0.989	17	3

agent_speech_fidelity: the real entity errors it now catches include an extra letter or digit in a code, a wrong digit count in an ID, a vowel substituted in a confirmation code, a letter said as a word, and a flight number said with the wrong value. On the review set, only 2% of turns change rating. Why: letters and digits in spelled codes must be spoken with the expected language's names, and ambiguous audio of an entity is an error.
tts_fidelity and user_speech_fidelity: the changes are few and mostly judge reasoning about missing or shifted words, for example audio that landed in the neighbouring turn. An entity substitution such as "EMP" spoken as "UMP" is now rated as an entity error.

English: stable on all three audio judges: one agent_speech_fidelity turn changed, and it also flips between the two Judge prompts: per-language guidance slot #249 runs. Two tts_fidelity turns changed, both about missing words.
Hindi: Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 has no Hindi guidance, so the shared wrong-language rules also flag the code-switching that is normal in Hindi, which lowers the Hindi faithfulness and agent_speech_fidelity scores. Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251 adds Hindi guidance that allows code-switching, which will change the scores for these two metrics.

Stack

Judge prompts: per-language guidance slot #249 Guidance slot and loader (base of this PR).
Judge prompts: shared rules for spelled codes, mixed-language speech, glosses and politeness #250 This PR: shared judge rules.
Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251 Per-language guidance content (based on this PR).

Versions are bumped in PR 3.

- The cultural-norms exception now also covers formal register and set service phrases. It allows one such phrase per function (acknowledge, hold, return from hold, thank, close), and a single short courtesy or empathy phrase is never filler on its own. - It does not cover repetition: the same formula twice in one turn, stacked acknowledgements, or a term given together with its synonym or translation are still filler. - Length is judged by how long the turn takes to say and how many information units it carries, not by written length. Polite constructions and numbers written out in words can look long without being long to hear.

- violating_policies: a phrase or clause in another language inside an otherwise correct-language turn ("one moment", "Let me check that") is part of "not communicating in the specified language". Established loanwords, codes, IDs, acronyms, product names and proper nouns are never a violation. - failing_to_disambiguate: a value that only contains letters or digits that sound alike is not by itself a flag. Phonetic confusability counts only when the trace shows the ambiguity mattered: a failed lookup, a user correction, two versions of the value, or a real homophone. - The fabricating_tool_parameters note on confusable letters now points to the language-specific guidance.

New violation 5: a term immediately followed by its translation, equivalent or abbreviation in parentheses or after a slash, such as "paid time off (PTO)" or "PIN / passcode". TTS does not voice the parentheses or slash, so the listener hears both forms back to back. A word, code or abbreviation used on its own is not a violation.

New block "Spelled codes and mixed-language speech": - Letters and digits in spelled codes must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) is wrong_language, plus entity_error if it sounds like a different letter or digit. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn is wrong_language, even when the rest of the turn is correct. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading is an error only if a native listener could take it for a different value.

New block "Spelled codes and mixed-language speech": - Letters and digits of trace entities must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) means the entity was not articulated correctly: rating 0. An entity that is itself in another language in the trace may be said in that language. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn: rating 0, quoting those words in the explanation. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading is rated 0 only if a native listener could take it for a different value.

New section "Spelled Codes and Mixed-Language Speech": - Letters and digits in spelled codes must be said with their names in the expected language. A letter or digit name from another language (the English "jay" or "six" in a non-English turn) rates at most 2 while the value is still unambiguous, and 1 if it sounds like a different letter or digit. - Spelling-alphabet words ("A as in Anna") are valid. - Confusable letters and digits are flagged only when the other one is clearly heard, not because of transcription or poor audio. - A phrase or sentence in another language inside the turn rates at most 2, and 1 if it replaces an entity or the turn no longer makes sense. - Proper nouns, product names, codes, acronyms and established loanwords are not wrong language. - An unusual but unambiguous number, date or time reading rates at most 2 if it sounds unnatural, and 1 only if a native listener could take it for a different value.

…hared-multilingual-rules

The shared rules in this branch change the English prompts of six judges. This records their new English renders.

- Expanding an acronym once, on its first mention ("your PTO, that's your paid time off"), is not repetition. A term given with its synonym or its translation into another language is still filler. The example no longer assumes a non-English call. - The one-phrase-per-function allowance also covers greet / open. - Polite constructions and numbers written out in words can look long without adding cognitive load. The previous wording said they don't take longer to say, which isn't true.

The rating scale had "briefly using the wrong language" at 2 and "repeatedly talking in a language not instructed" at 1, so a recurring English "one moment" in a French call could go either way. Now: - an isolated fixed phrase (a hold, acknowledgement or courtesy formula) is rated 2, even when it recurs across turns; - whole sentences of content in another language, or most turns in another language, are rated 1.

Violation 5 flagged "paid time off (PTO)", which is a normal acronym expansion in English HR and IT turns. It now covers only a term followed by its translation into another language or by a synonym that means exactly the same thing ("PIN / passcode"). Not a violation: - an acronym expanded once, on its first mention; - a slash between paired or alternative terms ("AM/PM", "and/or"); - a word, code or abbreviation used on its own. Also drop the claim that TTS never voices a slash; some engines do.

- Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error. The previous "benefit of the doubt" wording contradicted the entity_error definition. - A phrase in another language is an error only if it is not in the intended text; reproducing a phrase the intended text contains is not. - The wrong_language definition and the rating-0 definition now name the cases the new rules use: a phrase or clause in another language, and a letter or digit name from another language inside a spelled code. - An unusual reading covers only the intended words read in an unusual order or grouping (digit by digit vs as one number). A different word for the value ("septante" for "soixante-dix") is still an error, as the existing rule says. - The example no longer reads "a English turn" in English runs.

- A phrase in another language around the entities no longer rates the turn 0. Before, it did only when the turn had entities, so "One moment, je vérifie." was excluded while "One moment, je vérifie le code ZK3FFW." was rated 0 with a perfect code. Faithfulness and TTS fidelity already score the phrase. - A letter or digit name from another language still rates 0 when it is inside a trace entity. - Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error, as in the judge's own examples ("M" sounding like "N", "fifty" vs "fifteen"). - The example no longer reads "a English turn" in English runs.

- Confusable letters and digits: the rule now only stops the judge from flagging one because of its own transcription. Audio that is ambiguous between two letters or digits is still an error. The previous "benefit of the doubt" wording made the judge more lenient on the hardest cases, so fewer records with a mis-spoken code were rerun. - A phrase in another language is an error only if it is not in the intended text; reproducing a phrase the intended text contains is not. - An unusual reading covers only the intended words read in an unusual order or grouping. A different word for the value ("septante" for "soixante-dix") is still flagged, as the existing rule says. - The example no longer reads "a English turn" in English runs.

…hared-multilingual-rules

The fabricating_tool_parameters note on confusable letters pointed to "the language-specific guidance below". This branch adds no guidance, so the pointer moves to the branch that does.

sn-rachelhansen

I think this looks pretty good - I left a handful of comments. My only concern are the things I think are 'missing' (but these things may very well be covered in future PRs). The only thing that comes to mind immediately is the problem with speakability marking down things that are tags for the TTS model (ex. "[speaker cuts off]"). If I think of anything else, I'll leave send a message in the chat

Co-authored-by: sn-rachelhansen <rachel.hansen@servicenow.com>

fanny-riols · 2026-11-05T20:39:49Z

Thank you @sn-rachelhansen for your review! For speakability, I merged the fix last week :) #246

The bullet on unusual number readings ended with "is still an error, as above", pointing to the "Number surface forms" exception in the What to ignore list. Say the rule directly instead.

Same wording fix as for tts_fidelity: replace "as above" with the rule itself.

The conciseness suggestion and the two "different word" wording fixes change the English conciseness, tts_fidelity and user_speech_fidelity prompts on purpose.

Conciseness: a hold phrase stacked with another hold phrase or lead-in is repetition, like stacked acknowledgements. tts_fidelity, agent_speech_fidelity, user_speech_fidelity: a spelled code read as a word instead of letter by letter, or a letter in it replaced by an ordinary word, is an entity error; so is a name said without its diacritic or special letter so that it sounds different. These were listed for single languages in the per-language guidance but hold in any language.

…rd note for ko and hi Merge the latest shared judge rules. Dutch conciseness no longer lists lead-ins that announce the next step as filler. Korean and Hindi faithfulness now say, as Japanese does, that a transcript rendering a spoken loanword in Latin letters is not evidence of English speech.

s-rosen

Flagged a couple things. Apologies if I misunderstood the metrics

Speech fidelity (tts_fidelity, agent_speech_fidelity, user_speech_fidelity): remove the rule on names said without their diacritic, since a name's pronunciation often cannot be inferred from its letters; reword the letter-said-as-a-word case with an example. Conciseness: judge length from what the text shows (information units, runaway or stacked clauses) instead of how long the turn takes to say, which the judge cannot measure. Polite constructions and written-out numbers stay exempt.

…one information unit The shared conciseness prompt now judges length by information units, not by how long a turn takes to say, so the German entry no longer contrasts written and spoken length.

Co-authored-by: s-rosen <163597217+s-rosen@users.noreply.github.com>

A digit sequence that is normally read digit by digit (a PIN, phone number, confirmation or account number, or other code) but is read as one number is now an error even though the value is the same: rating 0 with entity_error in tts_fidelity, rating 0 in agent_speech_fidelity, and rating at most 2 in user_speech_fidelity.

…e correct The shared speech-fidelity prompts now treat a code or PIN read as one number as an error. French phone numbers are normally read in two-digit pairs, so the French entries say that this is correct.

… rule The code-or-PIN case is now an exception inside the existing rule on unusual number readings, instead of a separate bullet after it.

…e exceptions to the language rules

lindsaydbrin

Looks great overall! I just had a couple of comments from reading through. I also had Claude do a pass for redundancy and contradiction within each metric's judge prompt, because it was hard to keep track of which text lives in which prompt.

🤖 Thx, Claude.

…h guidance takes precedence

…ce count is a rough guide

…_fidelity checks entities only; user_speech_fidelity penalises any other-language speech; faithfulness rates an informative clause 1

gabegma · 2026-11-12T22:18:04Z

+        - Letters and digits in spelled codes must be spoken with their {expected_language} names. A letter or digit name from another language (e.g., an English letter or digit name in a German code) is rating = 0 with `wrong_language`; add `entity_error` if it sounds like a different {expected_language} letter or digit. If the intended text itself contains those foreign words or names, reproducing them is not an error.
+        - Spelling-alphabet words (e.g., "A as in Anna") are a valid way to say a letter.
+        - A spelled code read as a word instead of letter by letter (e.g., "INC" read as "ink"), or a letter in it said as an ordinary word instead of its letter name (e.g., Italian "J" said as "Gioia" instead of "i lunga"), is rating = 0 with `entity_error`.
+        - Confusable letters and digits: don't flag one because of your own transcription; in contrast, audio that is ambiguous between two letters or digits is still an error.


What does don't flag one because of your own transcription mean?

… rules in failing_to_disambiguate

… error, ignore and wrong_language sections; clarify sound-alike letters

… into the evaluation, low-fidelity and ignore sections; clarify sound-alike letters

…into Error Types and What to Ignore; clarify sound-alike letters

…, as for the other dimensions

gabegma · 2026-11-13T17:03:48Z

        - Phonetic spellings of Latin/Roman letters in non-Latin-script languages (e.g., Korean "에이" for "A", "비" for "B") — these are the standard spoken form of those letters in that language
        - Number surface forms that sound the same or differ only by minor grammatical agreement — accept the natural spoken rendering of a numeric value, including reversed digit order (German "vierundzwanzig" = 24), large-unit grouping (Japanese 万-based: "ichi-man" / 一万 = 10,000), and gender/agreement variants (Spanish "veintiún" vs "veintiuna"). Do NOT extend this to entirely different words for the same value (e.g., Swiss/Belgian "septante" vs standard French "soixante-dix" — these are different words and should be flagged).
        - Name spelling variants that are phonetically identical (e.g., German Meyer/Meier/Maier/Mayer) — only flag if the spoken sound clearly differs
+        - Spelling-alphabet words (e.g., "A as in Anna") used to say a letter


Why do we need to specify this? Shouldn't it be penalized if "A as in Anna" is in the intended text but not the audio?

gabegma · 2026-11-13T17:04:52Z

+        - A spelled code read as a word instead of letter by letter (e.g., "INC" read as "ink"), or a letter in it said as an ordinary word instead of its letter name (e.g., Italian "J" said as "Gioia" instead of "i lunga") — `entity_error`
+        - A code that is normally read digit by digit (a PIN, phone number, confirmation or account number) read as one number (e.g., PIN "1234" as "one thousand two hundred thirty-four"), or any reading that a native {expected_language} listener could take for a different value — `entity_error`
+        - Audio that could be either of two letters or digits that sound alike (e.g., "b"/"d") — `entity_error`


Could we explain a bit more the — entity_error`? I guess we mean this would count as an entity_error? Should we do this for the previous bullets too?

gabegma · 2026-11-13T17:29:36Z

+          - the language mismatch is clear and covers a meaningful portion of the turn;
+          - a phrase or sentence is in another language (e.g., an English "one moment" in a French turn), even when the rest of the turn is in {expected_language};
+          - a letter or digit in a spelled code or acronym is said with its name from another language (e.g., an English letter or digit name in a German code); add `entity_error` if it sounds like a different {expected_language} letter or digit. Codes and acronyms are not an exception: their letters and digits must be spoken with {expected_language} names.
+          Not wrong language: proper nouns, official product or system names, loanwords that are established in {expected_language}, and words in another language that are in the intended text.


I don't think we should penalize if the audio is in the wrong language, if the intended text also is (since this is already penalized by the faithfulness judge - that's not a TTS fidelity issue).

fanny-riols added 6 commits October 29, 2026 15:36

This was referenced Oct 29, 2026

Judge prompts: per-language guidance slot #249

Open

Judge prompts: guidance for de, nl, fr, fr-CA, es, it, hi, ko, ja #251

Open

fanny-riols added 11 commits October 29, 2026 16:32

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

537f5b3

…hared-multilingual-rules

Update the English golden prompts for the shared rules

57047b4

The shared rules in this branch change the English prompts of six judges. This records their new English renders.

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

1a816c3

…hared-multilingual-rules

Merge branch 'pr/fr/judge-language-guidance-infra' into pr/fr/judge-s…

2bffc63

…hared-multilingual-rules

Faithfulness: drop the pointer to language-specific guidance

20713cc

The fabricating_tool_parameters note on confusable letters pointed to "the language-specific guidance below". This branch adds no guidance, so the pointer moves to the branch that does.

fanny-riols marked this pull request as ready for review November 3, 2026 21:19

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

sn-rachelhansen reviewed Nov 4, 2026

View reviewed changes

fanny-riols and others added 2 commits November 5, 2026 15:38

Update configs/prompts/judge.yaml

cdae19b

Co-authored-by: sn-rachelhansen <rachel.hansen@servicenow.com>

Apply pre-commit

1ab32ab

fanny-riols added 3 commits November 5, 2026 15:46

TTS fidelity: state the "different word" rule in place

1779127

The bullet on unusual number readings ended with "is still an error, as above", pointing to the "Number surface forms" exception in the What to ignore list. Say the rule directly instead.

User speech fidelity: state the "different word" rule in place

a3aa6c4

Same wording fix as for tts_fidelity: replace "as above" with the rule itself.

Regenerate the English judge prompt golden

2672500

The conciseness suggestion and the two "different word" wording fixes change the English conciseness, tts_fidelity and user_speech_fidelity prompts on purpose.

fanny-riols and others added 2 commits November 6, 2026 15:09

Apply pre-commit

4b5a081

s-rosen reviewed Nov 6, 2026

View reviewed changes

fanny-riols and others added 3 commits November 9, 2026 11:52

Apply batched suggestions from code review

984b081

Co-authored-by: s-rosen <163597217+s-rosen@users.noreply.github.com>

Apply pre-commit

c3fa4aa

fanny-riols added 2 commits November 9, 2026 15:18

Speech fidelity: fold the digit-by-digit case into the number-reading…

1d27cf4

… rule The code-or-PIN case is now an exception inside the existing rule on unusual number readings, instead of a separate bullet after it.

Faithfulness and speech fidelity: language-specific guidance may stat…

177534d

…e exceptions to the language rules

lindsaydbrin reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

lindsaydbrin reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

lindsaydbrin approved these changes Nov 12, 2026

View reviewed changes

fanny-riols added 3 commits November 12, 2026 13:35

Speech fidelity and faithfulness: clarify confusable letters and whic…

158f35c

…h guidance takes precedence

Conciseness: one hold phrase with its lead-in is not stacking; senten…

70bd903

…ce count is a rough guide

Wrong language: codes and acronyms are not an exception; agent_speech…

f822772

…_fidelity checks entities only; user_speech_fidelity penalises any other-language speech; faithfulness rates an informative clause 1

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

Comment thread configs/prompts/judge.yaml Resolved

gabegma reviewed Nov 12, 2026

View reviewed changes

fanny-riols added 5 commits November 13, 2026 10:51

Faithfulness: wrong_language dimension; merge the sound-alike letters…

c7765c2

… rules in failing_to_disambiguate

tts_fidelity: move the spelled-code and mixed-language rules into the…

dd5b343

… error, ignore and wrong_language sections; clarify sound-alike letters

agent_speech_fidelity: move the spelled-code and mixed-language rules…

738f967

… into the evaluation, low-fidelity and ignore sections; clarify sound-alike letters

user_speech_fidelity: move the spelled-code and mixed-language rules …

c2f8063

…into Error Types and What to Ignore; clarify sound-alike letters

Faithfulness: drop the severity pointer from the wrong_language scope…

a767888

…, as for the other dimensions

gabegma reviewed Nov 13, 2026

View reviewed changes

Conversation

fanny-riols commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Validation

Stack

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sn-rachelhansen left a comment

Choose a reason for hiding this comment

Uh oh!

fanny-riols commented Nov 5, 2026

Uh oh!

s-rosen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lindsaydbrin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabegma Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

fanny-riols Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

gabegma Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

gabegma Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

gabegma Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fanny-riols commented Oct 29, 2026 •

edited

Loading