| --- |
| language: |
| - hi |
| - ne |
| license: apache-2.0 |
| tags: |
| - tokenizer |
| - sentencepiece |
| - devanagari |
| - hindi |
| - nepali |
| --- |
| |
| # LMA Phase 1 — Hindi and Nepali tokenizers |
|
|
| Two independent sentencepiece tokenizers, one per language, trained from |
| scratch for a pair of ~25M-parameter decoder-only Transformers. |
|
|
| **They share nothing** — not merges, not pieces, not a vocabulary file. Hindi |
| and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two |
| corpora and the two vocabularies separate is the central constraint of the |
| project rather than an afterthought. |
|
|
| | language | algorithm | vocab | trained on | corpus tokens | |
| |---|---|---|---|---| |
| | Hindi | unigram | 10,000 | 2,016,377 of 4,032,755 lines (50%, sampled at random) | 655.2M | |
| | Nepali | unigram | 10,000 | 2,539,821 of 5,079,643 lines (50%, sampled at random) | 558.9M | |
|
|
| ## How these were chosen |
|
|
| Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k) |
| across two algorithms (BPE, unigram), for both languages. Every model read the |
| same 10% random sample of its language's training split, so differences between |
| them are differences between models rather than between samples. |
|
|
| **Unigram beat BPE at every vocabulary size in both languages** — ten paired |
| comparisons, no exceptions, by 1.2–2.7% fertility. |
|
|
| **Vocabulary 10,000 was selected over 16,000** despite 16k tokenizing better. |
| The embedding matrix is `vocab_size × 512` parameters against a 25M budget, so |
| 16k spends 33% of the whole model on a lookup table while 10k spends 20%. The |
| selection rule is the smallest vocabulary whose fertility is within 8% of the |
| best, which trades ~5–7% fertility for ~3.1M parameters returned to the |
| transformer layers. |
|
|
| **Hindi** — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces. |
| **Nepali** — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces. |
|
|
| **UNK is impossible.** `byte_fallback=True` decomposes any unseen character |
| into byte tokens, so the UNK count is zero by construction rather than by luck. |
|
|
| ## Training data |
|
|
| | | Hindi | Nepali | |
| |---|---|---| |
| | final corpus | 5,094,185 docs / 2.588B chars | 6,755,888 docs / 2.513B chars | |
| | training split | 3,564,832 docs / 1.798B chars | 4,726,832 docs / 1.752B chars | |
|
|
| Splits are document-level, stratified by source, 70/15/15 by characters, with |
| one whole source held out per language as an unseen-domain test set. The |
| tokenizers saw the training split only. |
|
|
| The corpora are at |
| [meet5568/lma_datasets](https://huggingface.co/datasets/meet5568/lma_datasets). |
|
|
| ## Usage |
|
|
| ```python |
| import sentencepiece as spm |
| from huggingface_hub import hf_hub_download |
| |
| path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model") |
| sp = spm.SentencePieceProcessor(model_file=path) |
| print(sp.encode("भारत एक विशाल देश है।", out_type=str)) |
| ``` |
|
|
| ## Limitations |
|
|
| Unigram's memory scales with total corpus length — it builds a suffix array |
| over every character — at roughly 10.6 GB of RAM per GB of text, measured. The |
| full training split would need about 49 GB, so the final unigram models were |
| trained on 50% of it. A control experiment found fertility differing in the |
| fourth decimal place between 81.6% and 100% of the lines, so this is a hardware |
| limit rather than a quality one, but it is stated rather than implied. |
|
|
