---
language:
  - bg
  - bs
  - ca
  - cs
  - cy
  - da
  - de
  - el
  - en
  - es
  - et
  - eu
  - fi
  - fr
  - ga
  - gl
  - hr
  - hu
  - is
  - it
  - lb
  - lt
  - lv
  - mk
  - mt
  - nl
  - "no"
  - pl
  - pt
  - ro
  - ru
  - sk
  - sl
  - sq
  - sr
  - sv
  - tr
  - uk
tags:
  - tokenizer
  - sentencepiece
  - bpe
  - multilingual
  - european-languages
license: apache-2.0
---

# OpenEuroLLM Tokenizer (256k)

A **262,144-token SentencePiece BPE tokenizer** designed for efficient tokenization across all EU official languages and additional European languages. Trained on 173 GB of curated multilingual text from the OpenEuroLLM data catalogue on LUMI HPC.

## Key Results

- **Best fertility (tokens per word) on 26 out of 38 European languages**, outperforming Llama 3.2, Gemma 3, GPT-OSS, EuroLLM, Qwen 2.5, DeepSeek V3, and Mistral v0.3
- **Lowest average fertility overall: 2.12** across 38 languages
- Particularly strong on lower-resource EU languages: Lithuanian (31% fewer tokens than Llama), Hungarian (22%), Icelandic (26%), Maltese (25%)
- English fertility (1.58) within 5% of best (GPT-OSS, 1.51) — no trade-off

## Usage

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("openeurollm/tokenizer-256k")

text = "Hello world! Bonjour le monde. Hej världen!"
ids = tok(text)["input_ids"]
decoded = tok.decode(ids, skip_special_tokens=True)

print(f"Tokens: {len(ids)}")
print(f"Decoded: {decoded}")
```

### Batch encoding

```python
texts = [
    "The quick brown fox jumps over the lazy dog.",
    "Der schnelle braune Fuchs springt über den faulen Hund.",
    "Le rapide renard brun saute par-dessus le chien paresseux.",
]
batch = tok(texts, padding=True, return_tensors="pt")
print(batch["input_ids"].shape)  # (3, max_len)
```

### Special Tokens

| Token | ID | Purpose |
|-------|------|---------|
| `<unk>` | 0 | Unknown |
| `<bos>` | 1 | Beginning of sequence |
| `<eos>` | 2 | End of sequence |
| `<start_of_turn>` | 3 | Chat turn start |
| `<end_of_turn>` | 4 | Chat turn end |
| `<start_of_image>` | 5 | Image start |
| `<end_of_image>` | 6 | Image end |
| `<image_soft_token>` | 7 | Image placeholder |
| `<fim_prefix>` | 8 | Fill-in-middle prefix |
| `<fim_middle>` | 9 | Fill-in-middle middle |
| `<fim_suffix>` | 10 | Fill-in-middle suffix |
| `<tool_call>` | 11 | Tool call start |
| `</tool_call>` | 12 | Tool call end |
| `<unused_0>`–`<unused_99>` | 13–112 | Reserved for future use |
| `<pad>` | 262,144 | Padding |

## Training Details

| Parameter | Value |
|-----------|-------|
| **Algorithm** | BPE (SentencePiece) |
| **Vocabulary size** | 262,144 |
| **Training data** | 173 GB multilingual corpus |
| **Data mix** | 70% English, 10% code/math, 20% other languages (37 languages) |
| **Character coverage** | 0.9995 |
| **Normalization** | Identity (lossless) |
| **Byte fallback** | Enabled |
| **Digit splitting** | Enabled |
| **Max piece length** | 16 |
| **Trained on** | LUMI HPC (CSC, Finland) |
| **Training time** | ~9 hours (32 CPUs, 128 GB RAM) |

### Data Sources

The training corpus aggregates cleaned/deduplicated text from: C4, FineWeb-2, Nemotron-CC, MADLAD-400, HPLT, FinePDFs, German-Commons, StarCoder, Proof-Pile-2, Cosmopedia-v2, and FineMath.

### Languages (37 + English)

**EU Official (23):** bg, hr, cs, da, nl, et, fi, fr, de, el, hu, ga, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv

**Additional European (14):** sq, eu, bs, ca, gl, is, lb, mk, no, ru, sr, tr, uk, cy

## Fertility Evaluation

Average tokens per word across 38 European languages (lower = better), evaluated on 200 Wikipedia articles per language:

| Tokenizer | Vocab | Avg Fertility | Languages Won |
|-----------|-------|---------------|---------------|
| **Ours 262k** | **262k** | **2.12** | **26** |
| GPT-OSS 20B | 200k | 2.26 | 8 |
| EuroLLM 1.7B | 128k | 2.27 | 3 |
| Ours 128k | 131k | 2.31 | 0 |
| Gemma 3 4B | 262k | 2.35 | 0 |
| DeepSeek V3 | 129k | 2.52 | 0 |
| Llama 3.2 1B | 128k | 2.56 | 1 |
| Qwen 2.5 | 152k | 2.83 | 0 |
| Mistral v0.3 | 33k | 2.97 | 0 |

## See Also

- [openeurollm/tokenizer-128k](https://huggingface.co/openeurollm/tokenizer-128k) — 128k vocab variant (half the vocabulary, ~9% more tokens per word)