Tokenmaxxing BERT: 50x faster tokenization

Community Article

Published October 15, 2026

Upvote

Simon Brandeis

sbrandeis

huggingface

How skipping ASCII made BERT's tokenizer 50× faster

Over the past months we've been reviving the 🤗 tokenizers library. We aimed at faster encoding and a smaller footprint, and we are currently working towards stabilizing the improvements in a v1.0.0 release.

Check out the tokenizers v1 announcement post

Our north star was encoding throughput in MB/s of text for the tokenizers most LLMs use: BPE or Byte-Pair Encoding. Classic BPE tokenizers now encode up to 30× faster than in tokenizers 0.23 🔥🔥

While other model types got faster too, I was not satisfied with how we treated the good ol' bert-base-uncased, which now encodes only about 8.6× faster.

This post explains how we made BERT up to another 5.7× faster: about 50× faster than 0.23 in total. At the time of writing, the changes are still in Pull Request stage.

Figure 2: Speed-up in throughput from tokenizers v0.23

BERT? Really?

Why bother, you'd say? BERT was released in 2018, before ChatGPT (!!!). It should be irrelevant, right?...

Not quite. BERT-like models are everywhere. They're the encoders behind embeddings, search and RAG, rerankers and classifiers, and they're downloaded at a scale few LLMs reach. The three most downloaded models on the whole Hub all run BERT's normalizer: all-MiniLM-L6-v2 (218 million monthly downloads), the ms-marco-MiniLM-L6-v2 cross-encoder (79 million) and bge-small-en-v1.5 (59 million). bert-base-uncased itself is sixth, at 34 million.

Across the 10,000 most downloaded models on the Hub, BPE tokenizers (most LLMs) account for 842 million downloads a month. WordPiece, the tokenizer of BERT-like models, accounts for 627 million, and every one of those downloads goes through BERT's normalizer. Normalization isn't only a BERT thing, either: more than half of BPE downloads go through a Unicode normal form (mostly NFC, as in Qwen or ModernBERT), which this work speeds up too.

Figure 1: Monthly downloads of the most popular Hub models, by tokenizer algorithm, split by the normalizer their tokenizer runs.

Why so slow

When dealing with performance work, a very useful thing to do is to ✨ profile ✨ your software. So that's what we did: we profiled a warm encoding pass of English text with bert-base-uncased on 1.0. You can run the same profile yourself with our reproducer.

Here is the flame graph:

Figure 3: Bar width = share of encode time. Normalizer: 88%.

If that's the first time you ever encounter this chart, don't worry: I got you covered :)

How to read a flame graph

Imagine taking a photo of the program a thousand times a second. Each photo shows who is working, and for whom: "encode asked normalize for help, and normalize asked is_mark_nonspacing."

Pile up the photos and do a head count. A function that shows up in most of them is where most of the time went.

A flame graph draws that pile:

Each box is a function. The wider it is, the more photos it shows up in, so the more time it took.
Boxes sit on top of whoever called them. encode is at the bottom, because everything happens on its behalf.
Look for the widest boxes near the top. That's where the time really goes.

Left to right isn't time, just layout. Colours: yellow is the normalizer, blue the Unicode table lookups inside it, purple the pre-tokenizer, green WordPiece.

A few big takeaways from this chart:

88% of the encode is spent normalizing the text. WordPiece, the part that actually picks tokens, is 8%.
About two thirds of the whole encode resolves to a mysterious binary_search_by 🤔. It comes from char::is_mark_nonspacing() in the unicode_categories crate: the check for "is this character an accent?". We'll see why it's so expensive further down.

Okay, so now we know what's slow: text normalization. The point of this article is to explain how we made this normalizing step go brrrr.

Norma-what?

Bro wtf is text normalization

I hear you say behind your screen. I'll try my best at explaining, please bear with me!

Before a tokenizer splits text into tokens, it normalizes it: it rewrites the text into a canonical form, so that inputs that look the same to a human come out the same. Here's the whole journey of a sentence through bert-base-uncased's tokenizer, from text to the ids the model receives:

Figure 4: What bert-base-uncased does to the opening line of Proust’s Du côté de chez Swann, stage by stage.

Why would you ever want to do that??

If you wondered that, I'm sorry, but you'll have to read the next section. It's a bit heavy though, so I won't resent you if you skip it!

Unicode normalization in 3 minutes

Unicode is a way to represent text as numbers. Text is divided into code points: a letter a, a digit 5, an accènt, an ideogram 中, an emoji 🤗.
Each code point has a number from 0 to 0x10FFFF, about 1.1 million possible values. Unicode 17.0 assigns 159,801 of them.
Each code point is stored as 1 to 4 bytes. UTF-8, the encoding used almost everywhere, writes the 128 ASCII characters as a single byte with the high bit at 0, and everything else as 2 to 4 bytes with the high bit at 1. Remember that bit; it comes back later.

Figure 5: UTF-8, the encoding behind nearly all text on the web.

The same character on screen can have several representations. What a reader perceives as one character is a grapheme cluster, and it can be made of one or several code points. é can be a single code point, U+00E9, or two: e followed by U+0301, a combining acute accent.

Figure 6: Same glyph, two spellings. Normalization picks one.

é is a trivial case, unfortunately. A grapheme cluster can have many more parts:

Figure 7: What a reader sees as one character (a grapheme cluster) can be several code points: an accent, a skin tone, two flag letters, people joined by zero-width joiners.

That's an issue for language models: two strings that look identical to us would become different tokens. Hence normalization, which picks one spelling. Unicode defines four standard forms:

Figure 8: NFC composes, NFD decomposes; the K forms also fold compatibility characters (ligatures, circled digits) into plain ones.

BERT normalization

Great, normalization. Why does BERT spend 88% of its time there, then?

BertNormalizer (with lowercase=True, as in bert-base-uncased) does five things, in this order:

Clean: removes control characters and turns every whitespace character (tabs, newlines, Unicode spaces) into a plain space, or 0x20;
Chinese characters: puts a space on each side of every CJK ideogram;
NFD: decomposes every character (see the previous section for details)
Strip accents: drops the combining marks NFD split off;
Lowercase.

Figure 9: The five steps of the uncased BERT normalizer, used by bert-base-uncased, all-MiniLM and bge-en.

In 0.23 and 1.0, that's one character pipeline (tokenizers/tk-encode/src/normalizers/bert.rs on main, simplified):

input.chars()
    .filter(|&c| !clean_text_removes(c))              // is_control(c): a Unicode table lookup
    .map(clean_text_map)                              // whitespace → ' '
    .flat_map(pad_chinese_char)                       // ' ', c, ' '
    .nfd()                                            // unicode-normalization
    .filter(|c| !c.is_mark_nonspacing())              // strip accents: a Unicode table lookup
    .flat_map(char::to_lowercase)                     // a Unicode table lookup

For NFD and the accent check, it relies on external dependencies: unicode-normalization and unicode_categories. Here's what is_mark_nonspacing() does under the hood:

pub static MARK_NONSPACING: &'static [char] = &[
    '\u{0300}', '\u{0301}', '\u{0302}', '\u{0303}', '\u{0304}', '\u{0305}', '\u{0306}', '\u{0307}',
    // … 1,567 entries in total, one per character
];

fn table_binary_search(target: char, table: &'static [char]) -> bool {
    table.binary_search(&target).is_ok()
}

That's a binary search over a sorted array of 1,567 characters (6 KB): about 11 comparisons per lookup, each one a branch the CPU can't predict, jumping across memory.

And the 0.23 and 1.0 code ran lookups like this, for every. single. character. Including s, l, o and w!

How do we improve this, then?

Profiling your data

Profiling the code tells you where the time goes. Profiling the data tells you what work you can skip. Here are a few things about real Unicode text that will help us optimize the normalization step.

Most of the text is ASCII

ASCII is the first 128 code points, 0x00 to 0x7F: digits, punctuation, the usual whitespace, and the Latin alphabet without accents. In the texts we benchmark:

Figure 10: Share of bytes that are ASCII in the benchmark corpora: almost all of English and code, and still about a tenth of Arabic, Greek and Russian.

ASCII has interesting properties for us:

No normalization form changes an ASCII character, and an accent can't attach across one: consecutive ASCII characters are never reordered or altered by Unicode normalization.
ASCII is easy to lowercase. Capital and lowercase letters differ by a single bit:

Figure 11: In ASCII, a capital and its lowercase letter differ only in bit 5 (0x20): lowercasing is one OR.

ASCII is easy to find. In UTF-8, a byte is ASCII if and only if its high bit is 0. Super easy!

So we can just skip normalization on runs of ASCII characters. On English text, that alone removes almost all the work.

For very fast skips, we use the CPU's vector instructions (SIMD, "single instruction, multiple data") to check 16 bytes at once:

Figure 12: Is any high bit set? If not, the whole block is ASCII.

Apparté on SIMD

Modern CPUs often support operations on 128-bit registers holding 16 bytes, and instructions that work on all 16 bytes at once. The normalizer loads 16 bytes, gathers their 16 high bits into one integer with a single instruction, and checks it:

zero: all 16 bytes are ASCII, and the whole block is skipped: never decoded, never looked up, never copied;
not zero: one more instruction (count trailing zeros) gives the position of the first non-ASCII byte.

let mask = u8x16::from(chunk).to_bitmask();   // the 16 high bits, packed into one integer
if mask != 0 {
    return offset + mask.trailing_zeros() as usize;   // where the first non-ASCII byte is
}

About three instructions for 16 bytes, where the old code decoded and looked up each character one by one. BERT's ASCII-only fixes (whitespace, control characters, A–Z) are done the same way: comparisons on 16 bytes at once, then one OR with 0x20 on the capitals.

The rest of the text is clustered

Clustered in two ways:

A script is contiguous in code point space. Cyrillic, for example, is 256 consecutive code points, U+0400–U+04FF.
A stretch of text uses very few scripts. A Russian paragraph is Cyrillic plus ASCII spaces and punctuation; a French one is ASCII with a sprinkle of accented letters.

That makes a very good case for memoization (read: caching).

The only thing the normalizer needs to know about a character is whether normalization could touch it: change it, or merge it with its neighbour. So we compute that answer for a whole chunk of 256 code points the first time we meet one character from it, and keep the 256 yes/no answers. After that, any character from the page costs a single memory read instead of a binary search. Magic!

Figure 13: “Could normalization touch this character?”, answered once per page of 256 code points.

The locality of the text gives a high hit rate, and the locality of the cache in memory makes each hit fast: a chunk is four 64-byte cache lines, which stay in the fastest cache next to the core (see the appendix on CPU memory levels).

Most of the text is already normalized

Last but not least. Even the characters that could change usually don't: an NFC model mostly receives text that is already NFC.

Hence, we indulge in the art of "being lazy":

Skip fast: stretches with nothing to do are never copied;
Allocate memory once: if nothing changed, the input string is handed back as is, with no allocation. On the first real change, the output is allocated once and the untouched parts are copied in bulk.

Interactive demo

OK, that was a bit heavy. Here's an interactive demo that goes through the process step by step, on your own text, in your browser, so you can visualize what's going on before and after.

It runs the real Rust code compiled to WebAssembly.

Click here for the full-screen demo

Results

Good. We've seen why BERT is slow, what we can do to make it faster, and why. Let's now dive into the results!

Flamegraph

First, the updated flamegraph (see Why so slow for how to read one)

Figure 14: Same English text. Total width = encode time: main 23 MB/s, branch 151 MB/s.

Satisfying, right? Same text, same scale: the normalizer went from 88% of the encode to 13%.

End-to-end encoding speedups

BERT

Figure 15: bert-base-uncased, branch vs main, end to end. Token ids identical on every corpus.

The more ASCII, the bigger the speedup:

Figure 16: Each point is one corpus: end-to-end speedup over main against the share of its bytes that are ASCII.

English and source code gain the most, mostly from skipping ASCII ranges. Russian, Greek and Arabic are only about 11% ASCII but still gain 2.2–3.2×: those gains come from the page cache. Chinese doesn't gain at all; see the caveats below.

ModernBERT (NFC)

ModernBERT uses standard NFC normalization instead:

Figure 17: modernbert-base (NFC), branch vs main, end to end. Token ids identical on every corpus.

Here the gains come mostly from the cache. NFC on main already skipped ASCII, so English barely moves (1.06×), while code, Arabic and Russian gain about 2×.

Caveats

We added machinery, and very small inputs pay for it without much benefit. Some scripts, CJK for example, barely speed up: they have very little ASCII, and in BERT every Chinese character must be padded with spaces anyway. So they mostly pay for optimizations they don't benefit from.

Figure 18: Normalizer only, branch vs main. Slower beyond noise: 55 of 320 cells (19 of them at ≤ 32 B).

Tiny inputs (≤ 32 bytes) can be slower: down to 0.35× for StripAccents on 8 bytes of Chinese.
Cased BERT on Chinese is 8–15% slower from 64 bytes up.

Appendices

A. CPU memory levels and latency

A CPU doesn't read memory one byte at a time. It pulls 64-byte lines into small, fast caches next to the core. A load served from the L1 cache takes about a nanosecond; one that has to go all the way to RAM takes about 90.

Figure 19: Typical load latencies, to scale in time: while one load goes to RAM and back, about 90 hit L1.

That's why the page cache in The rest of the text is clustered is fast. A binary search over a 6 KB table jumps around memory, and every comparison is a branch the CPU can't predict. A page of 256 answers is four cache lines that stay in L1 while the text keeps hitting the same script.

B. Reproducing the measurements

Stating the obvious, but: The MB/s figures depend on your hardware. Across machines, but also on the same machine depending on what is already running there. Raw throughput numbers will change whether you run this or that CPU model, whether your RAM link is half-fried, whether you run the latest COD on ultra resolution while running the benchmark...

The speedup factors are more reliable. We measured on two AMD EPYC 7R13 hosts and an Intel Xeon Platinum 8488C (the markers on the charts), one thread each, and the speedup ratios agree within a few percent.

For reproducibility, I prepared and uploaded the necessary code in a GitHub repository.

Here's the full setup we used to get the numbers in this blog's chart:

Machines. We ran the benchmark code on Hugging Face Jobs: Two AMD EPYC 7R13 machines (cpu-performance flavor), along with one internal host running on Intel Xeon Platinum 8488C. Before trusting the numbers, we ran the same binary against itself to measure the machine noise: rounds varied by 0.35% on the EPYC hosts and about 2% on the Xeon.
What's compared. main branch of tokenizers (a few commits after 1.0.0-rc2, commit fb49a29) against the optimized branch (5b5a3fc, both built from source with the same toolchain (rustc 1.99.0), lto = "fat" and codegen-units = 1. For the releases chart, 0.23.2 comes from crates.io.
How. One thread, pinned to one core. Each measurement encodes a 5–6 MB corpus of text repeatedly in a loop for 5 seconds. One warm-up round, then 5 rounds in which main and the branch alternate, so slow drift hits both sides equally. We report the median of the per-round ratios. CPU time stolen by other tenants was 0.002% or less.
Same tokens out. Every run hashes the token ids it produces, and the hashes are identical across main, the branch and 0.23.2 on every corpus.
Corpora. hf-internal-testing/tokenizers-test-data at revision 5f5c652: English, Russian, Greek, Arabic and Chinese prose, plus mixed source code. The decomposed Greek corpus is the NFD form of the Greek one.
Input sizes. The normalizer-only sweep (Caveats) times 8 B to 64 KB slices, about 4 MB per cell, best of 3, over 4 rounds. A same-binary control sets the noise band (±3%).
Profiles. perf couldn't run in the job containers, so the flame graphs come from pprof-rs, sampling at 999 Hz inside the process. Instruction counts from callgrind back them up, with the cost of loading the tokenizer subtracted. On 1 MB of English, encoding drops from 482 million to 72 million instructions (6.7× fewer), and the normalizer's share drops from 87% to 12%.