---
language:
- hi
- ne
license: apache-2.0
tags:
- tokenizer
- sentencepiece
- devanagari
- hindi
- nepali
---

# LMA Phase 1 — Hindi and Nepali tokenizers

Two independent sentencepiece tokenizers, one per language, trained from
scratch for a pair of ~25M-parameter decoder-only Transformers.

**They share nothing** — not merges, not pieces, not a vocabulary file. Hindi
and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two
corpora and the two vocabularies separate is the central constraint of the
project rather than an afterthought.

| language | algorithm | vocab | trained on | corpus tokens |
|---|---|---|---|---|
| Hindi | unigram | 10,000 | 2,016,377 of 4,032,755 lines (50%, sampled at random) | 655.2M |
| Nepali | unigram | 10,000 | 2,539,821 of 5,079,643 lines (50%, sampled at random) | 558.9M |

## How these were chosen

Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k)
across two algorithms (BPE, unigram), for both languages. Every model read the
same 10% random sample of its language's training split, so differences between
them are differences between models rather than between samples.

**Unigram beat BPE at every vocabulary size in both languages** — ten paired
comparisons, no exceptions, by 1.2–2.7% fertility.

**Vocabulary 10,000 was selected over 16,000** despite 16k tokenizing better.
The embedding matrix is `vocab_size × 512` parameters against a 25M budget, so
16k spends 33% of the whole model on a lookup table while 10k spends 20%. The
selection rule is the smallest vocabulary whose fertility is within 8% of the
best, which trades ~5–7% fertility for ~3.1M parameters returned to the
transformer layers.

**Hindi** — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces.
**Nepali** — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces.

**UNK is impossible.** `byte_fallback=True` decomposes any unseen character
into byte tokens, so the UNK count is zero by construction rather than by luck.

## Training data

| | Hindi | Nepali |
|---|---|---|
| final corpus | 5,094,185 docs / 2.588B chars | 6,755,888 docs / 2.513B chars |
| training split | 3,564,832 docs / 1.798B chars | 4,726,832 docs / 1.752B chars |

Splits are document-level, stratified by source, 70/15/15 by characters, with
one whole source held out per language as an unseen-domain test set. The
tokenizers saw the training split only.

The corpora are at
[meet5568/lma_datasets](https://huggingface.co/datasets/meet5568/lma_datasets).

## Usage

```python
import sentencepiece as spm
from huggingface_hub import hf_hub_download

path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("भारत एक विशाल देश है।", out_type=str))
```

## Limitations

Unigram's memory scales with total corpus length — it builds a suffix array
over every character — at roughly 10.6 GB of RAM per GB of text, measured. The
full training split would need about 49 GB, so the final unigram models were
trained on 50% of it. A control experiment found fertility differing in the
fourth decimal place between 81.6% and 100% of the lines, so this is a hardware
limit rather than a quality one, but it is stated rather than implied.