pt-es-hi-tokenizer
A shared multilingual byte-level BPE tokenizer (GPT-2/Llama-3 style) for Portuguese, Spanish, and Hindi — one vocabulary, not per-language, intended for a single multilingual pre-training run rather than three separate models.
- Vocab size: 32,000, chosen via the Kneedle algorithm (point of maximum curvature on the compression-vs-vocab-size curve) over a candidate sweep of 8k/16k/32k/50k/65,536/100k, rather than a hand-picked threshold.
- Byte-level pre-tokenization: every input string is representable with no OOV/
<unk>fallback, which matters since the training corpus mixes Devanagari (Hindi) and Latin (Portuguese/Spanish) scripts. - Special tokens follow the Llama-3 chat convention (
<|begin_of_text|>,<|end_of_text|>,<|pad|>,<|start_header_id|>/<|end_header_id|>/<|eot_id|>), so the tokenizer is chat-fine-tuning-ready without resizing the embedding table later.
Training data
Trained on a MinHash-deduplicated corpus of pt-fineweb2, es-fineweb2, hi-fineweb2, and hi-sangraha (ai4bharat/sangraha's "verified" split), deduplicated per-language with datatrove (hi's two sources deduplicated against each other, not just internally).
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("andre15silva/pt-es-hi-tokenizer")
tok.encode("Olá, ¿cómo estás? नमस्ते")
Evaluation
Fertility (tokens/word, lower is better) and compression (bytes/token, higher is better) on a disjoint held-out sample:
| lang | fertility | compression |
|---|---|---|
| pt | 1.459 | 4.332 |
| es | 1.366 | 4.552 |
| hi | 3.247 | 3.951 |
| overall | 1.773 | 4.264 |
Hindi's fertility is notably higher than pt/es — consistent across every scale this was measured at (pilot, pre-dedup, and this final run), pointing to Hindi being comparatively data-starved in the training corpus rather than a tokenizer-vocab problem.