HuggingFace Tokenizers — быстрая токенизация текста

★ 7.7 · ai-tooling

huggingface-tokenizers is a Claude Code skill that brings high-performance text tokenization powered by a Rust core, capable of processing 1 GB of text in under 20 seconds — 10–100× faster than pure Python implementations. It supports three training algorithms: BPE (used by GPT-2, RoBERTa, BART), WordPiece (BERT, DistilBERT), and Unigram (ALBERT, T5, mBART), along with custom vocabulary training, alignment tracking from tokens back to original text positions, and batch encoding with padding and truncation. Dependencies include tokenizers, transformers, and datasets. The skill is aimed at NLP researchers and ML engineers who need to tokenize large corpora efficiently or train domain-specific tokenizers from scratch for production pipelines.