Tokenmaxxing BERT：分词速度提升 50 倍
原标题：Tokenmaxxing BERT: 50x faster tokenization
AI 摘要
Hugging Face 博客介绍了 tokenizers 库的优化工作，通过性能分析发现 BERT 分词器 88% 的编码时间花在文本规范化上，其中大部分消耗在 unicode_categories crate 的 char::is_mark_nonspacing() 检查。作者针对这一瓶颈进行优化，使 bert-base-uncased 的编码速度再提升 5.7 倍，相比 tokenizers 0.23 总计约快 50 倍。该优化对 Hub 上大量 BERT 类模型（如 all-MiniLM-L6-v2 等嵌入、检索、RAG 和分类模型）有广泛影响，目前改动仍处于 Pull Request 阶段。
正文节选
Over the past months we've been reviving the 🤗 tokenizers library. We aimed at faster encoding and a smaller footprint, and we are currently working towards stabilizing the improvements in a v1.0.0 release. Check out the tokenizers v1 announcement post Our north star was encoding throughput in MB/s of text for the tokenizers most LLMs use: BPE or Byte-Pair Encoding. Classic BPE tokenizers now encode up to 30× faster than in tokenizers 0.23 🔥🔥 While other model types got faster too, I was not sat