Skip to content

About

HuggingFace tokenizer.json in Zig 0.16: BPE, WordPiece and Unigram, differential-tested against the Rust tokenizers.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

tokenizers-zig

Status · Roadmap · Contributing · Issues · All SMC17 work

CI Release License: AGPL-3.0-or-later Zig

Zig 0.16 tokenizers compatible with the HuggingFace tokenizer.json format. Covers BPE (GPT-2 / Llama / Qwen / Mistral), WordPiece (BERT / DistilBERT), and Unigram (T5 / mBART / XLM-R) model families.

Current bounded task — 12 November 2026

The original 32-token WordPiece fixture now matches pinned Hugging Face tokenizers==0.21.4 on all 48 short inputs. The eight-input regression matched only two inputs before the punctuation repair; all eight now agree. A generated predicate observes the reference's behavior over all 1,112,064 Unicode scalar values: 726 punctuation scalars in 156 ranges. Currency symbols and some CJK letters no longer split as punctuation, while Arabic/Indic punctuation does.

zig build test --summary all
zig build -Doptimize=ReleaseSafe
python3 -m unittest discover -s tests -p test_diff_check.py
python3 tools/check_wordpiece.py

The current local gate passes 166 of 195 Zig tests; 29 real-model/fuzz tests remain skipped because their separate fixtures are absent. Seven Python gate fault tests pass. The offline task checks exact token IDs against frozen reference output; optional live verification uses the pinned wheel and fetches no models. See the original fixture contract.

Empty dumps, duplicate line IDs, malformed token IDs and matching errors now fail the comparator. Earlier code reported 0/0 empty dumps as success.

This task covers token IDs for one explicit vocabulary and the pinned BERT punctuation component. It does not certify BPE, Unigram, general normalization, offsets, every Encoding field, or current real-model parity. Existing fixture expectations remain unchanged; no release or visibility change is made.

Pinned real-model follow-up

Optional python3 tools/fetch_real_fixtures.py downloads three immutable, checksum-verified tokenizer JSON files (about 3.7 MB total; no weights). With those files, all 195 tests pass. The existing 39-input corpus now matches BERT 39/39 and TinyLlama 39/39 against the pinned reference. T5 remains 36/39; tools/check_real_parity.py --model all reports the mismatches and fails. The files stay gitignored; existing real-fixture expectations are unchanged. See the task contract for exact commands and source identities.

Historical status and remaining model gaps

v0.27 — full pipeline, all three model families, 6-subcommand tokenize CLI, batch-with-types APIs, HF Encoding return fields, sub-token offsets, property-fuzz-tested. The current clean-source result is above; 29 real-model tests (BERT / T5 / TinyLlama) skip because the HF tokenizer.json fixtures are not distributed here. To enable them, place the upstream tokenizer.json files in tests/fixtures/ under the names the tests expect (e.g. bert-base-uncased-tokenizer.json). The 600-iter invariant fuzz also requires those fixtures.

Differential correctness vs HF Rust — see COMPATIBILITY.md. An earlier write-up (parity_proof.pdf, June 2026, removed in November and kept in git history) claimed universal bit-identical parity; that claim was false — a 39-line adversarial corpus found real mismatches on all three model families. Root-caused and fixed 6 bugs: SentencePiece byte-fallback was unimplemented (TinyLlama → 39/39); Unigram never collapsed consecutive-unknown runs (T5 → 36/39); and BertNormalizer's lowercase was ASCII-only, its clean_text didn't strip zero-width joiners, its NFD table had zero Hiragana/Katakana decomposition entries, and NFD decomposition wasn't recursive (needed for Vietnamese's double-diacritic characters) (BERT → 38/39). The historical BERT currency gap is resolved in the current pinned replay. Three T5 gaps remain (confirmed cause: T5 uses SentencePiece's separate Precompiled-charsmap normalizer, entirely unimplemented — falls through to a silent no-op rather than erroring, because erroring breaks loading T5 at all) remain open. Full detail, reproduction steps, and the open gaps are in COMPATIBILITY.md — read that before the PDF.

Historical timing evidence

The older timing tables remain in STATUS.md. They have not been remeasured in this acceptance run and are not a current speedup claim. Compatibility failures must be resolved separately from timing a single successful input.

Real-tokenizer canonical IDs verified

All three model families have downloaded real-tokenizer fixture tests running on every zig build test (skipped if fixture absent):

  • bert-base-uncased (30522 vocab, WordPiece): hello=7592, world=2088, ,=1010, !=999. Added: [PAD]=0, [UNK]=100, [CLS]=101, [SEP]=102, [MASK]=103.
  • t5-base (32100 vocab, Unigram): ▁hello=21820. Added: =0, =1, =2, <extra_id_0>=32099, <extra_id_99>=32000.
  • TinyLlama-1.1B (32000 vocab, BPE+SentencePiece): "Hello, how are you today?" → [15043, 29892, 920, 526, 366, 9826, 29973].

What's covered

Models

  • BPE with priority-queue O(n log n) merge loop (vocab + merges; HF v1 string-merges and v2 array-merges both supported)
  • WordPiece (vocab + unk + ## continuation + max-input-chars guard; canonical BERT greedy longest-match with codepoint-aware boundaries)
  • Unigram (vocab + Viterbi DP best segmentation; single-codepoint unk fallback)

Pre-tokenizers

  • ByteLevel (canonical GPT-2 byte→unicode bijection, comptime static table)
  • Whitespace / WhitespaceSplit / BertPreTokenizer (whitespace + punctuation split for BERT word-by-word encoding)
  • Metaspace ("▁" replacement; reverse on decode)
  • Sequence (recursive composition)
  • Partial GPT-2 word-boundary regex pre-tokenize

Normalizers

  • Prepend + Replace + Sequence (Llama/Mistral SentencePiece chain)
  • BertNormalizer + ASCII lowercase
  • NFC / NFD / NFKC / NFKD Unicode normalization covering Latin, Latin Extended-A, Greek monotonic, Cyrillic precomposed, and algorithmic Hangul Jamo (U+AC00–U+D7A3)

Post-processing

  • TemplateProcessing wrap (BERT [CLS]…[SEP], T5 …</s>)
  • ByteFallback decode
  • Truncation + Padding
  • attention_mask + token_type_ids output (EncodeResult)
  • Pair templates for sentence-pair tasks (BERT NSP-style)

Batch APIs

  • encodeBatch(texts) → [][]u32 — sequential batch encode
  • encodeBatchParallel(texts) → [][]u32 — thread-pool parallel batch
  • encodeBatchWithTypes(texts) → []EncodeResult — sequential, full EncodeResult with type_ids + attention_mask per text
  • encodeBatchPairWithTypes(pairs_a, pairs_b) → []EncodeResult — pair batch (BERT NSP scale-out)

Loader + dispatch

  • HuggingFace tokenizer.json loader dispatching on model.type
  • Model-type inference for older HF tokenizer.json format (BERT / T5 omit model.type)
  • added_tokens end-to-end (parse → longest-match scan in encode → verbatim emit in decode)

Quickstart

const std = @import("std");
const tokenizers = @import("tokenizers");

pub fn main(init: std.process.Init) !void {
    const alloc = init.gpa;
    const io = init.io;

    const json = try std.Io.Dir.cwd().readFileAlloc(
        io, "tokenizer.json", alloc, std.Io.Limit.limited(64 * 1024 * 1024),
    );
    defer alloc.free(json);

    var tok = try tokenizers.fromJson(alloc, json);
    defer tok.deinit();

    // Simple encode (ids only)
    const ids = try tok.encode(alloc, "Hello, world!");
    defer alloc.free(ids);

    // Full EncodeResult with attention_mask + type_ids
    var result = try tok.encodeWithTypes(alloc, "Hello, world!");
    defer result.deinit(alloc);
    // result.ids, result.attention_mask, result.type_ids

    // Decode round-trip
    const back = try tok.decode(alloc, ids);
    defer alloc.free(back);
}

Same code works against BPE, WordPiece, or Unigram tokenizer.json — fromJson dispatches on model.type internally.

Build

zig build test                              # 195 tests (29 skip without HF fixtures)
zig build test-fuzz                          # standalone fuzz (BERT/T5/TinyLlama × 200 random texts)
zig build tokenize                          # build tokenize CLI (6 subcommands)
zig build bench                             # synthetic encode throughput
zig build bench-real-tinyllama              # real Llama tokenizer encode bench

CLI (tokenize)

tokenize encode      <tokenizer.json> "<text>"        # full pipeline encode
tokenize encode-base <tokenizer.json> "<text>"        # raw model output (no template)
tokenize encode-pair <tokenizer.json> "<a>" "<b>"     # BERT NSP-style pair
tokenize encode-full <tokenizer.json> "<text>"        # ids + type_ids + mask
tokenize decode      <tokenizer.json> "<ids,csv>"     # decode IDs → text
tokenize info        <tokenizer.json>                 # model / vocab / config

Siblings

Comparable upstream

  • huggingface/tokenizers — Rust canonical with Python + Node bindings. Widely deployed, hand-tuned, covers all model types. Reference for correctness; the per-family head-to-head benches above run against this.

License

AGPL-3.0-or-later. See LICENSE.

Honest non-claims

  • Pre-1.0 substrate per the Zig 0.16 ecosystem convention. Claims stay below release-readiness language until the language and stdlib hit 1.0.
  • Special-token attributes are partially honoured. special: true/false flag is used; single_word, lstrip, rstrip, normalized flags are parsed but not all enforced.
  • Regex / Split pre-tokenizers. GPT-2 word-boundary splitting is partial — modern Llama/Qwen tokenizers work correctly; raw GPT-2 / RoBERTa may diverge from HF on long mixed-whitespace inputs.
  • Tested on two platforms: x86_64 Linux and macOS on Apple silicon (fresh clone, 2026-11-12), Zig 0.16.
  • Synthetic-fixture WordPiece bench. The 5.3x ratio is measured against a 123-byte fixture; on longer documents the absolute ratio shifts because HF's fixed per-call overhead is amortised. Longer-input numbers are not measured yet.
  • Not bit-identical to HF Rust on all input. See COMPATIBILITY.md: on the 39-input adversarial corpus, BERT and TinyLlama match 39/39 and T5 matches 36/39. The 3 T5 gaps come from the unimplemented Precompiled normalizer (#11).

Reproducible setup

nix develop        # Zig 0.16.0 + jq + curl + gawk + python3(websockets)
zig build test     # 195 tests; 166 pass without HF fixtures, 29 skip

The repo's flake.nix pins exact dependency versions for nix develop.

Unified C ABI, Python Wheel & WASM

Core primitives from this repository are also cross-pollinated into SMC17/zinging (zig-substrate) with C11 ABI bindings (199 symbols), standalone Python wheels, JSON-RPC daemon (substrate serve), and wasm32 builds.

About

HuggingFace tokenizer.json in Zig 0.16: BPE, WordPiece and Unigram, differential-tested against the Rust tokenizers.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages