SentencePiece — языконезависимая токенизация текста

★ 7.7 · ai-tooling

sentencepiece is a Claude Code skill that provides language-independent tokenization of raw Unicode text using BPE and Unigram algorithms. It requires no pre-tokenization or language-specific preprocessing, making it well-suited for multilingual NLP pipelines and CJK languages (Chinese, Japanese, Korean), with throughput of 50,000 sentences per second and a memory footprint of roughly 6 MB. The skill exposes a Python API and CLI tools for training custom tokenizer models on raw corpora — a 100 MB corpus trains in 1–2 minutes — along with encode/decode operations and subword regularization for data augmentation. It underpins T5, ALBERT, and XLNet (Unigram) as well as mBART (BPE). Ideal for teams building reproducible, deterministic tokenization into multilingual or low-resource language models.