Model card

README.md

@@ -0,0 +1,84 @@
+---
+language:
+- hi
+- ne
+license: apache-2.0
+tags:
+- tokenizer
+- sentencepiece
+- devanagari
+- hindi
+- nepali
+---
+
+# LMA Phase 1 — Hindi and Nepali tokenizers
+
+Two independent sentencepiece tokenizers, one per language, trained from
+scratch for a pair of ~25M-parameter decoder-only Transformers.
+
+**They share nothing** — not merges, not pieces, not a vocabulary file. Hindi
+and Nepali both use the Devanagari block (U+0900–U+097F), so keeping the two
+corpora and the two vocabularies separate is the central constraint of the
+project rather than an afterthought.
+
+| language | algorithm | vocab | trained on | corpus tokens |
+|---|---|---|---|---|
+| Hindi | unigram | 10,000 | 2,016,377 of 4,032,755 lines (50%, sampled at random) | 655.2M |
+| Nepali | unigram | 10,000 | 2,539,821 of 5,079,643 lines (50%, sampled at random) | 558.9M |
+
+## How these were chosen
+
+Twenty models were compared: five vocabulary sizes (8k, 10k, 12k, 14k, 16k)
+across two algorithms (BPE, unigram), for both languages. Every model read the
+same 10% random sample of its language's training split, so differences between
+them are differences between models rather than between samples.
+
+**Unigram beat BPE at every vocabulary size in both languages** — ten paired
+comparisons, no exceptions, by 1.2–2.7% fertility.
+
+**Vocabulary 10,000 was selected over 16,000** despite 16k tokenizing better.
+The embedding matrix is `vocab_size × 512` parameters against a 25M budget, so
+16k spends 33% of the whole model on a lookup table while 10k spends 20%. The
+selection rule is the smallest vocabulary whose fertility is within 8% of the
+best, which trades ~5–7% fertility for ~3.1M parameters returned to the
+transformer layers.
+
+**Hindi** — fertility 1.3250 tokens/word, 3.7749 chars/token, 0 UNK, 0.289% byte-fallback, 201 unused pieces.
+**Nepali** — fertility 1.4620 tokens/word, 4.5208 chars/token, 0 UNK, 0.178% byte-fallback, 239 unused pieces.
+
+**UNK is impossible.** `byte_fallback=True` decomposes any unseen character
+into byte tokens, so the UNK count is zero by construction rather than by luck.
+
+## Training data
+
+| | Hindi | Nepali |
+|---|---|---|
+| final corpus | 5,094,185 docs / 2.588B chars | 6,755,888 docs / 2.513B chars |
+| training split | 3,564,832 docs / 1.798B chars | 4,726,832 docs / 1.752B chars |
+
+Splits are document-level, stratified by source, 70/15/15 by characters, with
+one whole source held out per language as an unseen-domain test set. The
+tokenizers saw the training split only.
+
+The corpora are at
+[meet5568/lma_datasets](https://huggingface.co/datasets/meet5568/lma_datasets).
+
+## Usage
+
+```python
+import sentencepiece as spm
+from huggingface_hub import hf_hub_download
+
+path = hf_hub_download("meet5568/lma_models", "tokenizer/hindi/hi_tokenizer.model")
+sp = spm.SentencePieceProcessor(model_file=path)
+print(sp.encode("भारत एक विशाल देश है।", out_type=str))
+```
+
+## Limitations
+
+Unigram's memory scales with total corpus length — it builds a suffix array
+over every character — at roughly 10.6 GB of RAM per GB of text, measured. The
+full training split would need about 49 GB, so the final unigram models were
+trained on 50% of it. A control experiment found fertility differing in the
+fourth decimal place between 81.6% and 100% of the lines, so this is a hardware
+limit rather than a quality one, but it is stated rather than implied.