File size: 3,406 Bytes
9f2ce12 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
language:
- hi
- ne
license: apache-2.0
tags:
- tokenizer
- sentencepiece
- devanagari
- hindi
- nepali
---
# LMA Phase 1 — Hindi and Nepali tokenizers
Two independent sentencepiece tokenizers, one per language, trained from
scratch for a pair of ~25M-parameter decoder-only Transformers.
**They share nothing** — not merges, not pieces, not a vocabulary file. Hindi
and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two
corpora and the two vocabularies separate is the central constraint of the
project rather than an afterthought.
| language | algorithm | vocab | trained on | corpus tokens |
|---|---|---|---|---|
| Hindi | unigram | 10,000 | 2,016,377 of 4,032,755 lines (50%, sampled at random) | 655.2M |
| Nepali | unigram | 10,000 | 2,539,821 of 5,079,643 lines (50%, sampled at random) | 558.9M |
## How these were chosen
Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k)
across two algorithms (BPE, unigram), for both languages. Every model read the
same 10% random sample of its language's training split, so differences between
them are differences between models rather than between samples.
**Unigram beat BPE at every vocabulary size in both languages** — ten paired
comparisons, no exceptions, by 1.2–2.7% fertility.
**Vocabulary 10,000 was selected over 16,000** despite 16k tokenizing better.
The embedding matrix is `vocab_size × 512` parameters against a 25M budget, so
16k spends 33% of the whole model on a lookup table while 10k spends 20%. The
selection rule is the smallest vocabulary whose fertility is within 8% of the
best, which trades ~5–7% fertility for ~3.1M parameters returned to the
transformer layers.
**Hindi** — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces.
**Nepali** — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces.
**UNK is impossible.** `byte_fallback=True` decomposes any unseen character
into byte tokens, so the UNK count is zero by construction rather than by luck.
## Training data
| | Hindi | Nepali |
|---|---|---|
| final corpus | 5,094,185 docs / 2.588B chars | 6,755,888 docs / 2.513B chars |
| training split | 3,564,832 docs / 1.798B chars | 4,726,832 docs / 1.752B chars |
Splits are document-level, stratified by source, 70/15/15 by characters, with
one whole source held out per language as an unseen-domain test set. The
tokenizers saw the training split only.
The corpora are at
[meet5568/lma_datasets](https://huggingface.co/datasets/meet5568/lma_datasets).
## Usage
```python
import sentencepiece as spm
from huggingface_hub import hf_hub_download
path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model")
sp = spm.SentencePieceProcessor(model_file=path)
print(sp.encode("भारत एक विशाल देश है।", out_type=str))
```
## Limitations
Unigram's memory scales with total corpus length — it builds a suffix array
over every character — at roughly 10.6 GB of RAM per GB of text, measured. The
full training split would need about 49 GB, so the final unigram models were
trained on 50% of it. A control experiment found fertility differing in the
fourth decimal place between 81.6% and 100% of the lines, so this is a hardware
limit rather than a quality one, but it is stated rather than implied.
|