Hugging Face Details tokenizers v1 Performance Overhaul and Architecture Changes

Hugging Face Details tokenizers v1 Performance Overhaul and Architecture Changes Hugging Face has published details on the release candidate for tokenizers v1, an architectural refactor designed to eliminate tokenizer bottlenecks in scaled machine learning workflows. While preserving outputs, APIs, vocabularies, and merge ranks from v0.23, v1 introduces core optimizations that yield 3 to 30 times faster single-threaded encoding across evaluated model families. v1 maintains identical token IDs, vocabularies, and API behavior to v0.23 while avoiding model specialization by preserving support for BPE, WordPiece, and Unigram. Single-thread encoding speeds on an Apple M4 Max improved by 3x to 30x across ten model families, scaling at 76% of linear performance across eight physical cores. Architectural changes include workspace splitting, SIMD-based bitstream pre-token splitting via bitcannon, a thread-local word cache, allocation-free scratch buffers, and rewritten BPE merge loops. Upcoming work for 1.0.0 targets unified training/inference encoding, optional metadata offsets, normalizer reworks, simplified Python bindings, and C/C++ bindings for inference runtimes. Architectural Optimizations in Encoding The performance gains in tokenizers v1 stem from targeted interventions across the pipeline. Instead of relying on a general-purpose regex engine, v1 uses 'bitcannon' to execute hand-written Boolean operations over bitstreams using CPU SIMD instructions to identify pre-token boundaries across 64 bytes per register operation. In the model stage, an intrusive doubly-linked list inside caller-owned scratch buffers replaces dynamic memory allocations and priority queues during BPE merges. Additionally, candidate merge pairs are packed into 64-bit integers with ranks in the high bits to allow branchless comparisons, and thread-local word caches memoize mappings from pre-token bytes to token IDs. Benchmarking Methodology and Scaling Measurements conducted via the tokbench repository enforce strict benchmarking rules: one identical timing loop across engines, excluding vocabulary loading from encode measurements, verifying output IDs via FNV-1a hashing, and pinning workers to distinct physical cores rather than SMT threads. Tests evaluated streams of distinct documents rather than repeatedly encoding a single cached document. On an Apple M4 Max, single-threaded encoding scaled from 3x faster on t5-base up to 30x faster on gpt2 compared to v0.23, maintaining 76% of linear scaling across eight workers through native parallelism that avoids single-lock contention. Workspace Modularization and Release Candidate Features The monolithic crate was split into a workspace where tk-encode serves as the runtime, while tk-serialize, tk-convert, and tk-train are linked only when required. The current pre-release candidate on crates.io also adds FlatCache, MPHF RankStore, BucketVocabStore, batched pre-token model calls, reusable decoding buffers that avoid intermediate string copies, and Node.js bindings. Training features are separated behind a default feature flag pulling a C++ dependency, which users can disable if only encode functionality is needed. Roadmap for 1.0.0 and Future Exploration Prior to the official 1.0.0 release, Hugging Face plans to port additional model families to the new merge loop, use tk-encode for validation during training, make token offsets and masks optional, rework normalizers with bitnorm support, and simplify Python bindings to better support free-threaded CPython. Inference-only C and C++ bindings are also planned for ExecuTorch and llama.cpp. Beyond 1.0.0, the team plans to explore tok-devices, an optional component investigating direct GPU encoding and batch decoding for large workloads.

Hugging Face 发布 tokenizers v1 候选版：编码性能最高提升 30 倍 Hugging Face 宣布推出 tokenizers v1 候选版（RC）。针对现代大模型训练与高并发场景中分词器可能成为性能瓶颈的问题，v1 在完全保持输出 ID、API 与词表兼容的前提下重构了执行路径，单线程编码吞吐提升 3 至 30 倍，并在 8 物理核上实现线性 76% 的扩展效率。 在 Apple M4 Max 单线程测试中，v1 覆盖的 10 个模型家族编码速度比 v0.23 快 3 到 30 倍（从 t5-base 到 gpt2），多线程 8 核扩展度达 76% 的线性性能。 保持与 v0.23 相同的输出 ID、API、词表和合并层级，单一 crate 拆分为 tk-encode、tk-serialize、tk-convert 和 tk-train 等子 workspace 以精简依赖。 引入 bitcannon 技术，将固定正则分割替换为基于 SIMD 指令集的 64 字节位流布尔运算，覆盖 GPT-2、cl100k、o200k 等主流模式。 设计线程局部 WordCache 缓存已合并的 pre-token，重写 BPE 合并循环以完全消除运行时堆内存分配并复用 scratch buffer。 制定严格的基准测试标准排除冷启动偏差；1.0.0 正式版将推进更多模型覆盖与 Python 绑定精简，后续计划探索 GPU 分词 tok-devices。 性能瓶颈与架构拆分 在大规模数据训练与高并发服务中，CPU 分词延迟可能导致 GPU 等待。tokenizers v1 在确保与 v0.23 输出 token ID 完全一致的前提下重写底层。原有的单一 crate 被拆分为多个工作区构件，应用仅需链接作为运行时的 tk-encode 即可，无需强制引入序列化或训练依赖，从而减小二进制体积。 核心加速技术：bitcannon 与 WordCache 针对 BPE 模型的 pre-token 正则分割，v1 使用针对固定语法手写的 bitcannon 函数替代通用正则引擎，利用 CPU 的 SIMD 指令在 64 字节寄存器上以位流布尔操作确定边界。对于实际文本中频繁重复的词汇，v1 新增线程局部 WordCache 映射，已处理过的 pre-token 字节可直接返回对应 ID 列表跳过合并计算。 无分配合并循环与原生并行 在 BPE 合并阶段，旧版本每次调用均分配新内存并构建优先队列，而 v1 将候选符号构建为扁平数组内的侵入式双向链表，所有临时状态存放在调用方提供的 scratch buffer 中，完全消除了运行时的内存分配开销。候选对与 rank 被打包为 64 位整型进行无分支比对，同时单次模型调用可批处理多个 pre-token，各线程独立持有缓存与缓冲池以避免锁竞争。 测试标准与路线图规划 评测基于 tokbench 运行，通过独立进程、物理核心绑定、哈希校验输出 ID 以及使用无法完全塞入缓存的真实不同文档语料以避免虚假命中。v1 RC 版目前已发布至 crates.io。在 1.0.0 正式版发布前，团队将统一训练校验与推理的编码实现、支持可选 offsets/masks、简化 Python 绑定并提供 C/C++ 推理绑定；1.0.0 之后将探索把词表上传至 GPU 实现端到端批处理解码与编码的 tok-devices。

The tokenizer has not historically been the bottleneck within ML workflows. Compute-wise, tokenization is light compared to the heavy modeling happening in the rest of the pipeline. Yet, in some cases, it has rapidly become key to accelerating (or slowing down) your machine learning work. As models become faster and workloads scale, that balance begins to shift. Training on massive datasets, serving many concurrent requests, or repeatedly processing long inputs can put enough pressure on the tokenizer that it starves the model of data. This is why we have chosen to heavily focus on performance for the upcoming version 1 of tokenizers. Tokenization should be light and should scale with your workflow. Your GPUs should never sit idle waiting for the CPU to complete its tokenization. In this article, we look at what makes v1 faster than v0.23, often by tens of times. This work was entirely possible thanks to the rest of the ecosystem. Tokenization is a very active area of open source work, and libraries such as gigatoken , tiktoken , kitoken , tokie , fastokens , wordchipper and ai-tokenizer , as well as many others, have each pushed on what a fast tokenizer can be. We read that work, and several of the ideas below reached us because another project showed they were worth trying. Before this refactor, tokenizers was nowhere near the performance it could have had, so contributing to it may not have seemed worth it. With this refactor, we hope to make clear that we intend tokenizers to be a library worth contributing to. We also thank IBM, NVIDIA, and the ExecuTorch team for contributing patches and helping us test across a wide range of hardware to broaden platform support. Results We showcase results for the release candidate of tokenizers v1 against other widely used alternatives. We go over single-threaded, multi-threaded, scaling across threads, per-model comparison, per-language comparison, latency, decoding throughput, memory heap, as well as crate size. We run this from the tokbench repository, and add a command to rerun the benchmarks on your hardware if you would like to do so. What V1 Is v1 will produce the same token IDs as v0.23. The goal was to preserve the output, the API, the vocabulary and the merge ranks, and improve everything that can be improved. That includes breadth. The library stays general across tokenizer families rather than specialising on BPE, so v1 loads everything v0.23 loaded. A tokenizer converts text into the list of integers a model reads. tokenizers runs that conversion in four stages. Normalization applies operations such as lowercasing or Unicode normalization to the raw text. Pre-tokenization splits the text into smaller pieces called pre-tokens. The model turns each pre-token into tokens and maps them to IDs in its vocabulary. Post-processing adds any special tokens the model expects. The model stage is where most of the work described here happens. Eight of the ten model families measured in this article use byte pair encoding, or BPE. BPE starts from the bytes of a pre-token and repeatedly joins the highest ranked adjacent pair until no ranked pair remains. The ranking is learned when the tokenizer is trained and ships with it, so the same text always produces the same IDs. A merge never crosses a pre-token boundary. The other two families use WordPiece and Unigram, the two other model types the library supports. The tokenization pipeline page documents the four stages. Tokenization algorithms documents BPE, WordPiece and Unigram. Each stage was worked on. These are the changes that mattered: change what it does workspace split one crate became a workspace: tk-encode is the required runtime, and tk-serialize , tk-convert and tk-train are linked only when an application needs them no-alloc model the merge working set lives in a caller-owned scratch buffer; the loop never touches the allocator bitcannon the split pattern becomes Boolean operations over bitstreams, using SIMD instructions to find splits instead of a regex engine merge-loop rewrite the pieces being merged form an intrusive doubly-linked list inside one preallocated buffer, so a merge updates two indices instead of moving data word cache a thread-local memo from pre-token bytes to finished ids, so a repeated word is merged once native parallelism one shared tokenizer encodes from many threads at once; each thread draws its scratch buffer and word cache from its own sub-pool, so threads no longer queue on a single lock ( #2365 ) The Split: Bitstreams Instead Of A Regex BPE models use a regular expression to split the input text into smaller, easier to process chunks called pre-tokens. Merges happen inside a pre-token and never across the boundary between two of them, so this split decides what the rest of the pipeline sees. That regular expression is a fixed parameter of the model. It ships with the tokenizer and never changes at runtime, so there is no need for a general-purpose regex engine to interpret it on every encode. An equivalent splitting function can be written by hand, once, for the pattern a given model actually uses. A hand-written function can then use the SIMD instructions (single instruction, multiple data) of a modern CPU, which apply one operation to many bytes at once and suit UTF-8 text well. bitcannon views the input's bytes as parallel streams of bits, so boundaries fall out of boolean operations across whole registers instead of a scan that advances one character at a time. It decides 64 bytes per register operation. The same idea drives Parabix for text processing and simdjson for JSON. This depends on recognising the pattern. A handful of grammars cover most byte-level BPE models, and a tokenizer whose pattern is not among them keeps the regex path and none of this speed-up. That is why the gains above vary as much as they do. The Word Cache Real text contains many repeated words. Because BPE always produces the same token IDs for a given pre-token, v1 can save the result after processing it once. A thread-local cache maps each pre-token's bytes to its token IDs, allowing later occurrences to skip the merge process. Naturally, as the input grows, the number of unique words can grow more slowly than the total number of words. Repeated words then account for an increasing share of the input. New words still appear, which accounts for the occasional misses in the animation below. Reproduce the shared-prefix result with: tokbench measure prefix-sharing \ --engine pipeline \ --engine hf-tokenizers \ --compare-to pipeline-no-cache \ --corpus agentic_swe Caching works best when the input contains repeated pre-tokens. Input with few repeated pre-tokens can pay for lookups without receiving many hits. The Merge Loop The next major cost comes from the BPE merge loop. For each pre-token, the loop repeatedly finds the highest-priority adjacent pair and merges it. The previous implementation allocated new memory for every call and built a new priority queue for every pre-token. v1 reuses a scratch buffer owned by the caller, removing those repeated allocations. It stores symbols in a flat array and links adjacent symbols by their positions in that array, which makes updates during merging cheaper. It also processes a batch of pre-tokens in a single model call. Each candidate pair is also packed into a single 64-bit value, with the merge rank in the high bits. Comparing two candidates is then just comparing two integers, and "no merge here" is the largest possible value, so the loop finds its next merge without a branch. Method Small differences in benchmark design can produce large differences in tokenizer performance. We used the following rules to keep the comparison consistent across engines. rule why one timing loop every engine runs the identical loop; no per-engine fast path load excluded vocabulary load is timed separately, never inside encode id-hash verified FNV-1a over the output ids must match the baseline exactly common cells only medians are over cells every engine ran and verified complete sweep per process each repeat starts in a new process and retains every cell physical-core pinning workers are pinned to eight distinct physical cores, never sibling SMT threads independent Jobs separate Jobs measure host-to-host variation Repeatedly encoding one document can be faster than encoding a stream of distinct documents on the same build. The first approach measures performance when the entire document is already represented in the cache. The second measures performance on new input while allowing previously seen pre-tokens to remain cached. Both conditions are sometimes described as "warm," even though they measure different workloads. Our headline results use distinct documents, and the complete corpus is too large to fit in the cache. Tokenizer benchmarks should identify which workload they use because the choice can dominate the result. What This Adds Up To Across the ten model families v1's encode path covers, it encodes text 3 to 30 times faster than v0.23 with one thread on an Apple M4 Max. The low end is t5-base, the high end gpt2. It scales at 76% of linear across eight workers. Throughout these changes, v1 produces exactly the same token IDs as the released library. The overall improvement comes from several changes working together: a hand-written splitter in place of a regex engine, a cache that answers a repeated word without merging it again, a merge loop that never touches the allocator, and one model call per batch of pre-tokens instead of one per pre-token. Each reduces the work done at a different point in the pipeline. The next priority is support for more model families. We will move additional models onto the new merge loop before 1.0.0 . Once the release candidates stabilize, the next step will be bringing about the improvements within the transformers library and the rest of the ecosystem which depend on the tokenizers library. This post is generated from tokbench results and will be updated as support expands. Getting It A release candidate for v1 is on crates.io. The API you call is the one you already call, so the only thing that changes is which build you install. It is the ordinary install: cargo add tokenizers --pre Training is behind a default-on feature that pulls a C++ dependency with it. If you only need to encode, turn it off to exclude the training implementation: cargo add tokenizers --pre --no-default-features --features http Encoding is unchanged: same call, same ids. use tokenizers::tokenizer::{ Result , Tokenizer}; fn main () -> Result <()> { let tokenizer = Tokenizer:: from_pretrained ( "deepseek-ai/DeepSeek-V4-Flash" , None )?; let encoding = tokenizer. encode ( "The tokenizer is no longer the bottleneck." , false )?; println! ( "{:?}" , encoding. get_ids ()); // [671, 17840, 9160, 344, 1119, 5827, 270, 111127, 16] println!("{:?}", encoding.get_tokens()); // ["The", "Ġtoken", "izer", "Ġis", "Ġno", "Ġlonger", "Ġthe", "Ġbottleneck", "."] Ok (()) } ``` For a batch, `encode_batch` is what scales across cores. It is the call the scaling view above measures. ```rust let encodings = tokenizer. encode_batch (documents, false )?; Every figure in this post was measured against this crate. The Python bindings wrap the same code and are built from bindings/python , but they add per-call overhead that none of these measurements include. Progress Towards V1 The benchmarks in this post cover the completed release-candidate work listed first. The remaining sections show what is still required for 1.0.0 and what we plan to explore afterward. Release Candidate: Implemented This work is in the Rust pre-release on crates.io: cargo add tokenizers --pre workspace split: divide the single crate into tk-encode , tk-serialize , tk-convert and tk-train , so an application links only what it uses bitcannon: replace regex splitting on the encoding path with bitstream operations covering GPT-2, cl100k, o200k, Tekken and DeepSeek. This replaced the finite-state machines that shipped first #2201 #2317 WordCache: reuse the token IDs of previously processed pre-tokens #2262 , af5a3e3 faster lookup and merging structures: add FlatCache, MPHF RankStore, incremental merging, and BucketVocabStore #2190 #2188 reusable model memory: move temporary model state into scratch buffers so tokenization does not allocate on each call #2175 #2183 pipeline post-processing: expose post-processing as the STAGE_POST pipeline stage #2182 batched model calls: process multiple pre-token spans in one call #2304 faster decoding: write decoded bytes directly into a reusable buffer, avoid intermediate strings and copies, accelerate token lookup, support buffered streaming, and decode batches in parallel role_to_token support #2343 Node.js bindings #2281 1.0.0 one encoding implementation: use tk-encode during training validation so training and inference cannot produce different tokenization results optional offsets and masks: compute this metadata only when requested, keeping it off the token-ID-only path rework normalizers bitnorm support, building on atomnorm #2209 spm precompiled simpler Python bindings: reduce locking, wrapper types, and handwritten dispatch code while preserving subclassing, serialization, custom decoders, mutation behavior, and support for free-threaded CPython inference-only C and C++ bindings for ExecuTorch and llama.cpp, with possible JVM, Swift, and Go bindings to follow After 1.0.0 tok-devices: explore GPU encoding and batch decoding while keeping text and token IDs on the device. The decoder would upload the vocabulary once, calculate output positions in parallel, and gather the corresponding bytes on the GPU. This would be an optional component intended for large batches, subject to further prototyping and measurement.