---
license: odc-by
language: multilingual
tags: [tokenizer, bpe, multilingual, vocabulary-scaling, cwt]
---
# CWT Multilingual BPE Tokenizer Ladder (32k / 64k / 128k / 256k)

Byte-level BPE tokenizers at four vocabulary sizes, trained on a **deterministic, reproducible**
multilingual-including-English corpus: **FineWeb v1 (English)** + **FineWeb-2 (non-English)** — the FineWeb-2
authors' own recommended way to get full coverage including English. Built for controlled **vocabulary-scaling**
studies where a 128k–256k vocab must be *genuinely occupied* (not English dead-tokens).

## Occupancy / fertility (held-out)
| vocab | occupancy | dead tokens | fertility (subwords/word) |
|---|---|---|---|
| 32,000 | 0.997 | 92 | 2.54 |
| 64,000 | 0.995 | 330 | 2.28 |
| 128,000 | 0.983 | 2,183 | 2.07 |
| 256,000 | 0.951 | 12,500 | 1.91 |

## Corpus
- English: `HuggingFaceFW/fineweb` (sample-10BT), rev `9bb295ddab0e`
- Non-English (28 langs): `HuggingFaceFW/fineweb-2`, rev `af9c13333eb9`
- English share: 40%; total 200,000 docs. Languages: arb_Arab, ben_Beng, ces_Latn, cmn_Hani, dan_Latn, deu_Latn, ell_Grek, fas_Arab, fin_Latn, fra_Latn, heb_Hebr, hin_Deva, hun_Latn, ind_Latn, ita_Latn, jpn_Jpan, kor_Hang, nld_Latn, pol_Latn, por_Latn, ron_Latn, rus_Cyrl, spa_Latn, swe_Latn, tha_Thai, tur_Latn, ukr_Cyrl, vie_Latn
- **Reproducible corpus:** [DaveGabe/cwt-multilingual-pretrain-mix](https://huggingface.co/datasets/DaveGabe/cwt-multilingual-pretrain-mix) (no RNG; first-N of pinned revisions).

## Use
```python
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast(tokenizer_file="fineweb_mix_256k/tokenizer.json")
```