Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
-
Updated
Aug 7, 2024 - Python
Unsupervised Word Segmentation for Neural Machine Translation and Text Generation
Unsupervised text tokenizer focused on computational efficiency
Fast and customizable text tokenization library with BPE and SentencePiece support
Cevahir Zihin: a mind that learns by living, not a Transformer. It keeps knowledge as evidence on ties, attends with a Seven-Place Focus and learns from every conversation. Also: a Turkish BPE tokenizer, a Transformer language-model core and cognitive systems.
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
Explains nlp building blocks in a simple manner.
Fast bare-bones BPE for modern tokenizer training
Build LLM from scratch
exact tokenizers ids, as fast as the hardware allows
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
Machine Learning for Phishing Website Detection
Accelerando-8192 — an 8,192-token alien glyph alphabet, interactive encoder, and complete visual atlas.
Subword Encoding in Lattice LSTM for Chinese Word Segmentation
To associate your repository with the bpe topic, visit your repo's landing page and select "manage topics."