README.md

---
language:
  - en
  - de
  - fr
  - es
  - pt
  - it
  - nl
  - pl
  - ro
  - cs
  - sv
  - da
  - "no"
  - fi
  - hu
  - hr
  - bg
  - tr
  - ca
  - ru
  - uk
  - sr
  - zh
  - ja
  - ko
  - ar
  - fa
  - he
  - hi
  - bn
  - th
  - vi
  - ka
  - hy
  - el
  - yi
  - ur
  - ta
  - te
  - gu
  - pa
  - ml
  - kn
  - am
  - si
  - my
  - km
  - mr
  - ne
  - or
  - bo
  - dv
  - eu
  - gl
  - gd
  - et
  - sk
  - lt
  - sl
  - lv
  - af
  - sq
  - sw
  - is
  - tl
  - cy
  - ga
  - br
  - la
  - mk
  - id
license: apache-2.0
library_name: tokenizers
tags:
  - tokenizer
  - bpe
  - multilingual
  - quartz
  - aenea
  - flores
pipeline_tag: text-generation
---

# QT_V.2 64K — Multilingual BPE Tokenizer

**The most equitable 64K tokenizer available.** 71 natural languages across 26 script families, with half the vocabulary of Llama 3, Tekken, and Qwen 2.5 — yet fewer total tokens on both FLORES-200 (204 languages) and our 66-test field benchmark.

Part of the **QT_V.2 tokenizer family** by [Quartz Data Infrastructure](https://quartz.host), the open data layer behind [AENEA](https://aenea.app).

## FLORES-200 Results (204 Languages · 1,012 Parallel Sentences)

| Metric | QT 64K | QT 96K | QT Code 114K | Llama 3 (128K) | Tekken (131K) | Qwen 2.5 (152K) |
|---|---|---|---|---|---|---|
| **Total tokens** | 13,592,357 | **12,961,617** | 13,007,924 | 16,764,198 | 14,421,539 | 15,425,680 |
| **Equity ratio** | **41.0×** | **31.6×** | 43.3× | 118.6× | 127.9× | 77.7× |
| Mean fertility | 4.18 | 3.94 | 4.03 | 5.72 | 5.34 | 4.91 |

The equity ratio measures the gap between best-served and worst-served language (lower is fairer). QT 64K at 41.0× is **2.9× more equitable than Llama 3** (118.6×) and **3.1× more equitable than Tekken** (127.9×) — at half the vocabulary.

### Where QT 64K Dominates (FLORES-200 tok/word)

| Language | QT 64K | Llama 3 | Tekken | Qwen 2.5 |
|---|---|---|---|---|
| Tibetan | **42.5** | 149.8 | 168.4 | 98.0 |
| Odia | **4.16** | 16.90 | 18.30 | 13.65 |
| Khmer | **17.1** | 40.9 | 70.5 | 30.7 |
| Georgian | **3.83** | 15.47 | 3.93 | 8.33 |
| Sinhala | **3.84** | 11.37 | 16.60 | 9.17 |
| Amharic | **3.90** | 11.95 | 11.98 | 6.45 |

## Field Benchmark (66 Tests)

| Metric | Value |
|---|---|
| **Total tokens** | **3,593** |
| vs Llama 3 (128K) | 36.3% fewer tokens |
| vs Tekken (131K) | 17.3% fewer tokens |
| vs Qwen 2.5 (152K) | 30.7% fewer tokens |

## When to Use This Variant

**QT_V.2 64K** is ideal when you need the smallest possible embedding table — for parameter-constrained small models, edge deployment, or when every MB of VRAM matters.

Also available: [QT_V.2 96K](https://huggingface.co/QuartzOpen/QT_V.2_96K) (best all-round) · [QT_V.2 Code 114K](https://huggingface.co/QuartzOpen/QT_V.2_Code_114K) (multilingual coding)

## Usage

```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
encoded = tok.encode("The quick brown fox jumps over the lazy dog")
print(encoded.tokens)
```

## Specifications

| Spec | Value |
|---|---|
| Vocabulary | 64,000 |
| Languages | 71 natural + 14 code |
| Script families | 26 |
| Pretokenizer | Llama 3 regex |
| Arithmetic | Single-digit splitting |
| Max token length | 15 chars |
| Avg token length | 5.7 chars |

## Training

Byte-level BPE with Llama 3 regex pretokenizer. Corpus: 58.5% Wikipedia (71 languages via wiki_ultra_clean v7.3), 18.0% code (14 languages), 23.5% Stack Exchange (49 sites via se_ultra_clean v1).

## Files

`tokenizer.json` · `vocab.json` · `merges.txt` · `training_report.json`

## Contact

Open-source: quartzopensource@gmail.com
Commercial licensing & enterprise: commercial@aeneaglobal.com

## License

Apache 2.0 — Copyright 2025-2026 AENEA Global Ltd

```bibtex
@misc{qt_v2_2026,
  title={QT_V.2: A Multilingual BPE Tokenizer Family},
  author={AENEA Global Ltd},
  year={2026},
  url={https://quartz.host},
}
```