Phase 1 Tokenizers — Hindi and Nepali
Two independent SentencePiece tokenizers trained from scratch for a pair of monolingual ~25M-parameter decoder-only language models. They share no training text, no vocabulary and no merges — the separation is what makes a Hindi/Nepali comparison meaningful.
Chosen models
| Language | Algorithm | Vocab | Fertility (tok/word) | chars/token | UNK rate |
|---|---|---|---|---|---|
| Hindi | BPE | 24,000 | 1.2542 | 4.1039 | 0.0e+00 |
Each was selected from a sweep of BPE and Unigram × 16,000 and 24,000, then
retrained on the complete cleaned corpus. tokenizer_sweep.json in each
language folder holds the full comparison.
Why these vocabulary sizes
Fertility improves monotonically with vocabulary, so it cannot be the only
criterion. At d_model=384 the embedding table costs vocab × 384 out of a ~25M
budget, and one decoder layer costs ~`12 × 384²` = 1.77M:
| Vocab | Embeddings | Share of budget | Decoder layers affordable |
|---|---|---|---|
| 16,000 | 6.14M | 25% | ~10.7 |
| 24,000 | 9.22M | 37% | ~8.9 |
| 64,000 | 24.58M | 98% | ~0 |
A vocabulary much above ~32k spends the model on its own embedding table.
UNK rate is zero by construction
byte_fallback=True: any character the vocabulary cannot cover decomposes into
UTF-8 byte tokens (ids 3–258). Nothing is ever discarded.
⚠️ Input must be normalized first
These were trained on text normalized by the project's common/normalize.py,
whose behaviour is language-specific: chandrabindu (ँ) folds to anusvara (ं)
for Hindi but not for Nepali, where the nasalization is phonemic.
Feeding un-normalized text to the Hindi model triggers byte fallback and inflates the token count. Always normalize with the matching language setting before encoding.
Usage
import sentencepiece as spm
from huggingface_hub import hf_hub_download
path = hf_hub_download("Prateek-Tiwari10/lma_phase1-tokenizer", "hindi/hindi_final.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("प्रधानमंत्री ने कहा कि सरकार काम कर रही है।", out_type=str))
Files
hindi/ hindi_final.model · hindi_final.vocab
tokenizer_final.json · tokenizer_sweep.json · sweep_models/
nepali/ nepali_final.model · nepali_final.vocab
tokenizer_final.json · tokenizer_sweep.json · sweep_models/