Skip to content

FEAT: Add PinyinConverter for Chinese Pinyin-mix transformations - #2649

Merged
Roman Lutz (romanlutz) merged 8 commits into
microsoft:mainfrom
Nanduu24:feat/pinyin-converter
Sep 15, 2026
Merged

Roman Lutz (romanlutz) merged 8 commits into
microsoft:mainfrom
Nanduu24:feat/pinyin-converter

Conversation

@Nanduu24

@Nanduu24 Nantha kumar (Nanduu24) commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Description

Adds PinyinConverter, a non-LLM text-to-text converter that rewrites selected Chinese (Hanzi) characters as Pinyin or initials. It supports Chinese-specific Pinyin-mix transformations described in CSSBench without translating the prompt into another language.

Behavior

  • mode: full produces toneless Pinyin, initial produces the first letter of each reading, and mixed independently chooses full Pinyin or an initial for each selected character.
  • Phrase-aware pronunciation: readings are resolved from the complete original prompt before replacements are applied. Unselected characters still provide context, so partial conversion does not lose the pronunciation of words such as 银行 or 重庆.
  • Han-character coverage: includes (ideographic zero), CJK Unified Ideographs Extensions A-J, and compatibility ideographs. Hanzi without a reading in the installed dictionary remain unchanged.
  • proportion ([0.0, 1.0]): selects round(proportion * number_of_hanzi) positions. Unselected Hanzi remain unchanged. Zero selected characters, including counts that round to zero, leave the prompt unchanged.
  • separator: inserted after converted characters, except at the end of the prompt. Original trailing spaces, newlines, and punctuation are preserved rather than trimmed.
  • seed: provides reproducible position selection and mixed-mode rendering through the converter's scoped RNG.
  • Original non-Hanzi characters are preserved. Prompts without Hanzi are unchanged.

Examples:

Settings Input Output
Defaults (mode="full") 银行 yinhang
Defaults (mode="full") 二〇二六 erlingerliu
mode="initial" 银行 yh
separator=" " 重庆 chong qing
proportion=0.0, separator="!" 你好! 你好!

Dependency and async behavior

pypinyin>=0.55.0 is a regular dependency, not a separate optional extra. It is MIT-licensed, pure Python, and adds no runtime transitive dependencies on PyRIT's supported Python versions. The minimum version supplies the extended-Hanzi readings covered by the regression tests. uv.lock resolves version 0.55.0 with artifact hashes.

The import remains lazy: importing PyRIT or constructing PinyinConverter does not load the Pinyin dictionaries. Dictionary loading and phrase lookup run in a worker thread through asyncio.to_thread, keeping synchronous dictionary I/O off the event loop. Conversion tests do not skip based on optional extras.

Documentation

Adds full-Pinyin and initial-letter examples to the text-to-text converter notebook and its paired Python source, including the actual output. Refreshes the converter modality reference table in the overview notebook. Converter-specific behavior is documented in the class docstring.

Testing

  • 60 Pinyin converter test cases cover modes, proportions, seeded reproducibility, validation, identifiers, phrase context, partial and mixed conversions, mixed-script input, preservation of original trailing characters, ideographic zero, extended CJK block boundaries, missing dictionary readings, event-loop responsiveness, and lookup-error propagation.
  • Adds Pinyin cases to the shared seeded-converter tests and dedicated coverage for explicit seed overrides and serial/parallel call-order independence in all three modes.
  • Adds lazy-import regression coverage to ensure resolving the public converter export does not load pypinyin.
  • Targeted converter, seeded-RNG, registry, lazy-import, and documentation-contract run: 417 passed, using the exact minimum dependency version 0.55.0.
  • Ruff checks, formatting checks, type checks, and applicable pre-commit hooks pass locally.
  • Verified that the notebook pair is synchronized, the saved Pinyin output matches execution, and existing outputs are preserved.

Closes #2647

Adds a deterministic (no-LLM) text converter that rewrites Chinese (Hanzi)
characters as their Pinyin romanization, a Chinese-specific adversarial pattern
described in recent Chinese LLM safety work (e.g. CSSBench). Pinyin mixing can
bypass keyword/token-level safety filters that match on Hanzi rather than on
romanized readings.

Supports three rendering modes (full reading, first-letter initial, and a
per-character mix), a configurable proportion of Hanzi to convert (values below
1.0 leave the rest as Hanzi, producing mixed Hanzi/Pinyin text), an optional
syllable separator, and a seed for reproducible selection. Non-Hanzi characters
always pass through unchanged.

pypinyin is added as an optional dependency (pip install pyrit[pinyin]) and
imported lazily, following the pattern used by other optional-dependency
converters. Tests that exercise conversion are skipped when pypinyin is absent.

Closes microsoft#2647
@Nanduu24

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve readings against the complete prompt before selecting replacements. Preserve original trailing characters when inserting separators and add regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Include Unicode 17 CJK extensions G-J in Hanzi selection and preserve characters without dictionary readings. Cover romanization, block boundaries, and partial-selection counts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds PinyinConverter to doc/code/converters/1_text_to_text_converters.py with a
Chinese-prompt example (full readings and first-letter initials), mirroring how
the script-specific Arabic converters are documented. Satisfies
test_all_converters_are_documented, which requires every converter to appear in
a converter notebook.
Offload dictionary lookup, require pypinyin 0.55.0, synchronize the notebook examples and outputs, and extend async and seeded-converter regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Sep 15, 2026
Merged via the queue into microsoft:main with commit 0884936 Sep 15, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Pinyin Mix converter for Chinese adversarial text transformations

2 participants