AksharaLP-Sentence-2.8M

It doesn’t guess one word at a time. It fixes the whole sentence.

Romanized Hindi → Devanagari, decided sentence by sentence, in 10.85 MB, on the device.
A word model proposes the spellings; a 2.77M-parameter neural Hindi language model picks between them jointly.

---

The one-line pitch. Hindi typed in English letters usually goes wrong at the sentence level, not the word level: hu should be हूं here and हुआ there, hai should be है or हैं, pata should be पता or पाता. AksharaLP Sentence keeps the word model's proposals and adds a tiny neural language model that reads the whole sentence before choosing — and it is still small enough to ship inside a web page.

_{Recorded from the live demo: Boht ache h Bhai, mai ghr pe hu is typed, decoded by the word decoder, then re-decoded by the sentence decoder in a Web Worker in the browser tab. Both decoders are in this repository.}

The numbers, up front

26.73% whole sentences exactly right Dakshina Hindi sentence test, 4,976 sentences	+1.02 pts sentences right vs the word model 95% CI 0.40–1.65 · significant	+0.42 pts words right vs the word model 95% CI 0.33–0.51 · significant
4.07 → 3.93 CER on the same 4,976 sentences WER 11.20 → 10.78	132,299 output words audited 0 outside their own candidate list	2.77M parameters in the sentence LM + 989K in the word model underneath

_{Every number above was measured by us, in a single one-shot run of the shipped configuration. v2 and v3 were scored with the same harness on the same sentences; intervals are paired bootstrap over sentences (2,000 resamples, seed 0). Settings were chosen on validation and internal dev sets only.}

Same sentence, two decoders

Real outputs, produced by running both decoders in this repository on the inputs below (the 21-sentence casual set in cases/, plus a few more of the same kind). Nothing here is illustrative: these are the exact strings the runtime returns.

You type	Word mode (v2)	Sentence mode (v3)
mai ghr pe hu	मैं घर पे हुआ	मैं घर पे हूं	fixed
mai school me hu	मैं स्कूल में हुआ	मैं स्कूल में हूं	fixed
mujhe nahi pata	मुझे नहीं पाता	मुझे नहीं पता	fixed
kal milte hai	कल मिलते है	कल मिलते हैं	fixed
hum ghr ja rhe hai	हम घर जा रहे है	हम घर जा रहे हैं	fixed
yaar mai thak gya hu	यार मैं तक गया हूं	यार मैं थक गया हूं	fixed
Boht ache h Bhai	बोहत अच्छे है भाई	बोहत अच्छे है भाई	not fixed (see below)
boht accha laga	बोहत अच्छा लगा	बोहत अच्छा लगा	not fixed (see below)
mein bazaar ja rahi hu	में बाजार जा रही हूं	में बाजार जा रही हूं	not fixed (see below)
nahi pata	नहीं पाता	नहीं पाता	not fixed (see below)
main ghar ja raha hoon	मैं घर जा रहा हूं	मैं घर जा रहा हूं	already right
wo ladki school gayi thi	वो लड़की स्कूल गई थी	वो लड़की स्कूल गई थी	already right
tum kya kar rahe ho	तुम क्या कर रहे हो	तुम क्या कर रहे हो	already right
ye kitab bahut acchi hai	ये किताब बहुत अच्छी है	ये किताब बहुत अच्छी है	already right
wo ghar me hai	वो घर में है	वो घर में है	already right

_{Word mode = the v2 decoder (word model + dictionary pack + bigram Viterbi). Sentence mode = the same candidates re-scored by the neural LM (search3, beam 8). Both ran on the same machine, same files, same harness. Boht → बोहत is a channel/candidate problem, not a sentence problem: standard spelling is बहुत.}

The four cases it does not fix, and why

Input	What comes out	Why the sentence LM cannot help
Boht ache h Bhai	बोहत अच्छे है भाई	बहुत is candidate #11 for boht — channel score −5.83, below the top-6 the LM is allowed to see (nlm_k 6, nlm_margin 2). The LM can only reorder what it is given, so the channel, not the LM, has to be fixed.
mein bazaar ja rahi hu	में बाजार जा रही हूं	मैं is candidate #2 at −2.14, just outside nlm_margin 2 (the LM is skipped for a slot when the top candidate leads by more than 2). हूं is right; the pronoun stays में.
boht accha laga	बोहत अच्छा लगा	Same बहुत/बोहत channel problem as the first row. अच्छा is already correct here, so there is nothing else for the LM to change.
nahi pata	नहीं पाता	No context. Two words, no auxiliary, no agreement: पाता is a perfectly good sentence ending in a bigger corpus than the two-word one. mujhe nahi pata — three words — is fixed to मुझे नहीं पता, which is exactly the point: the LM needs a sentence to have a sentence to read.

And one it does not get credit for: kal → कल. The older bigram LM already prefers कल (−1.03) to काल (−0.70) in every sentence we tried, so the neural LM has nothing left to do there.

Measured results

Set	Sentences	Word % v2 → v3	Sentence % v2 → v3	Δ sentence (95% CI)
Dakshina Hindi sentence test (held-out)	4,976	89.02 → 89.44	25.70 → 26.73	+1.02 [0.40, 1.65] ✓
Context set (ambiguous words in context)	66	94.89 → 97.02	86.36 → 90.91	+4.55 [0.00, 10.61] (words +2.13 ✓)
val600 (selection)	600	88.80 → 89.35	23.83 → 26.33	+2.50 [0.67, 4.33] ✓
val2 (selection-side)	1,000	89.33 → 89.91	25.60 → 26.60	+1.00 [−0.50, 2.50]
devA (casual chat, tuning)	488	92.37 → 93.82	84.22 → 87.30	+3.07 [1.23, 4.92] ✓
devB (casual chat, report-only)	480	91.54 → 92.22	82.50 → 83.96	+1.46 [−0.42, 3.54] not significant

Word-side intervals on the test set: +0.42 [0.33, 0.51]; sentence-exact: +1.02 [0.40, 1.65]. CER 4.07 → 3.93, WER 11.20 → 10.78. Word accuracy on the casual sets is token accuracy and sentence accuracy is whole-sentence exact match, both after Unicode NFC normalisation.

_{The test set was run once, after all settings were fixed. val600, val2 and devA are selection/tuning sets and are shown for completeness — their gains are real but they are not held-out. devB is the honest one: the sentence model gains 1.46 points of sentence accuracy there and that gain is not significant, which is why the card leads with the test-set number and not with devB.}

Word mode is unchanged, by construction

Benchmark	v2	v3
Aksharantar Hindi test (9,991 words), top-1	70.04	70.04
top-3	88.31	88.31
top-8	93.63	93.63

The sentence model overrides only the search over an existing candidate lattice; candidate generation is the v2 code path, unmodified. These are the v2 numbers, not a re-run — that is the point. (Not re-run for v3 because there is nothing different to run.)

What it changes

Gains and regressions on the 480-sentence casual-chat set (word accuracy per spelling pattern):

Improves	v2 → v3		Regresses	v2 → v3
Word-final y / h (y_end)	82.98 → 89.36 (+6.4)		Extra h (asp_add)	85.45 → 81.82 (−3.6)
Missing nasal (nasal)	86.00 → 90.00 (+4.0)		Stretched letters (repeat)	82.14 → 78.57 (−3.6)
Missing h in kh/th/bh (asp_drop)	71.19 → 74.58 (+3.4)		Dropped or added a (schwa)	97.44 → 94.87 (−2.6)
Chat abbreviations (abbr)	91.67 → 93.75 (+2.1)		Long / short vowel spelling (vlen)	96.43 → 94.64 (−1.8)
Extra doubled letters (gem_add)	98.39 → 100.00 (+1.6)		English words in chat (english)	98.39 → 96.77 (−1.6)
Ambiguous short words (homograph)	93.59 → 94.87 (+1.3)

The LM is good exactly where a bigram cannot be: verb endings, auxiliaries and agreement across the whole sentence. It is slightly too eager to normalise chat spellings toward dictionary forms — an extra h, stretched letters, a dropped or added a — and those rows are the cost. They are shown because they are real.

The guardrail

✓ It reorders candidates. It never invents words.

Every Devanagari word in the output is a candidate the word model already produced for that typed word. The sentence search can only choose between those candidates. Punctuation, digits and kept-as-Latin tokens pass through untouched. Each result carries guard: {ok, words, outside}, and the demo re-checks it on every keystroke.

Across every evaluation set — 132,299 output words — 0 fell outside their slot's candidate list.

That property is what makes this safe to ship: the sentence LM cannot hallucinate a spelling, drop a word, or reorder the sentence. Worst case, it picks a different candidate from the same short list the word model already offered.

Size

Part	On disk	gzip -9
Word model package (v2: xlit.q4 + params.json + dictionary pack + runtime)	16,259,506 B	7,919,816 B (7.92 MB)
nlm.q8 — sentence LM, 2,770,752 parameters, tokenizer embedded	3,239,640 B	2,921,848 B
nlm.mjs — LM loader + inference	8,430 B	3,132 B
v3.mjs — lattice beam search	4,942 B	2,115 B
Add-on total	3,253,012 B	2,927,095 B (2.93 MB)
Complete sentence package	19,512,518 B	10,846,911 B (10.85 MB)
Same system with the 4-bit LM (nlm.q4)	1,862,360 B	1,537,849 B → 9.46 MB complete

So: 10.85 MB complete with the 8-bit LM, of which the word-only model is 7.92 MB; 9.46 MB if you take the 4-bit LM. (nlm.q4e8 sits between the two at 2,184,860 B gz.) The 4-bit LM is within tuning noise and saves 1.39 MB: on the same tuning row (w_nlm 0.3, margin 4) it scores 89.45 val / 93.73 devA / 90.01 val2 against 89.44 / 93.82 / 89.95 for the 8-bit one, i.e. everything inside ±0.1 pt. The 8-bit LM ships because it is the file the final one-shot evaluation was run with. Every byte count above was measured with du -b and gzip -9.

Why the LM is this big (and not bigger)

The shipped LM is the Kaggle S model: 2,770,752 parameters, 21k steps, valid cross-entropy 4.188 nats/token (4.758 nats/word). A larger M model (3,766,080 parameters, 4.063 nats/token, 4.648 nats/word) was trained and evaluated too, and it was no better at the sentence task while being 1.3× bigger — so S ships. The LM is deliberately only asked to rank about six spellings per word, not to generate text, which is why 2.77M parameters are enough.

• Data: IndicCorp v2 Hindi, CC0. No scraped chat logs, no non-commercial corpora.
• Tokenizer: our own 8,000-piece unigram set, Viterbi segmentation reimplemented so Python and JavaScript tokenize identically.
• Model: 2-layer LSTMP (512 hidden, 160-dim embedding, tied input/output embedding, dropout 0.05), 8-bit quantised to 3.1 MB.
• Scoring: trained with a self-normalisation penalty, so the runtime can skip the softmax normaliser (norm: "self") — that is the fast setting used here (95 ms/sentence in JS vs 193 ms for exact normalisation).
• Dependency-free inference: nlm.mjs is plain JavaScript on Float32Array, no ONNX, no WASM, no GPU.

Speed

Path	Per sentence	Notes
Python reference, search only	13.6 ms	lattice given; on a shared box
JavaScript (Node), margin 2, norm self — the shipped setting	95 ms	200 sentences, box shared with other jobs
JavaScript (Node), margin 2, exact normalisation	193 ms	2× slower, ~+0.1 pt on tuning
JavaScript (Node), margin 4, norm self	406 ms	4.3× slower for −0.02 val / −0.10 devA tokens

_{These JavaScript numbers are deliberately pessimistic: they were measured on a shared machine while other research jobs were running, one sentence at a time, cold caches. A quiet machine, a warm candidate cache and a shorter sentence are all faster. Browser latency in the demo is shown live per sentence, in your tab, on your hardware — trust that number over this table.}

Usage

Browser (ES modules, no build step)

Download the files and serve them yourself — the Hub returns .mjs as text/plain, so a cross-origin import straight from huggingface.co/.../resolve/main/ is rejected by the browser's MIME check. Any static server that sends text/javascript for .mjs works (python3 -m http.server in the folder below does).

huggingface-cli download sourojitdotcom/AksharaLP-Sentence-2.8M --local-dir aklp-sent
cd aklp-sent && python3 -m http.server 8000     # then open http://localhost:8000/

<script type="module">
  import { AksharaLP } from "/aksharalp.mjs";
  import { attachSentence } from "/sent.mjs";

  const aklp = await AksharaLP.load("/", { dict: true });            // word package, 7.92 MB gz
  attachSentence(aklp, await (await fetch("/nlm.q8")).arrayBuffer()); // sentence LM, +2.93 MB gz

  const r = aklp.transliterate("mai ghr pe hu");
  console.log(r.output);        // मैं घर पे हूं
  console.log(r.guard);         // {ok: true, words: 4, outside: 0}

  aklp.sentenceMode = false;    // the same engine, word mode: मैं घर पे हुआ
</script>

sent.mjs is six dozen lines: it builds the v3 lattice from the word model's candidates and calls search3(lat, p, coreLm, nlm) from v3.mjs. Read it before you change it — the guardrail lives there. Run it in a Web Worker, as the demo does, so the page stays at 60 fps. If you only want to try it in a browser, use the demo Space — it is this exact bundle, already served with the right MIME types.

Node.js (≥ 18)

huggingface-cli download sourojitdotcom/AksharaLP-Sentence-2.8M --local-dir aksharalp-sent

import fs from "node:fs";
import { loadFromDir } from "./aksharalp-sent/aksharalp.mjs";
import { attachSentence } from "./aksharalp-sent/sent.mjs";

const aklp = await loadFromDir("./aksharalp-sent/");
attachSentence(aklp, fs.readFileSync("./aksharalp-sent/nlm.q8"));
console.log(aklp.transliterate("kal milte hai").output);   // कल मिलते हैं

Python (reference implementation)

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("sourojitdotcom/AksharaLP-Sentence-2.8M")
sys.path.insert(0, f"{path}/python")          # needs numpy + torch (CPU)
from bhc.translit3 import BhcTranslit3
from bhc.nlm import NLM

nlm = NLM.load(f"{path}/nlm.q8", norm="self")
xl = BhcTranslit3(path, nlm=nlm, w_nlm=0.4, w_lm=0.25, w_ch_s=1.5, w_agn=2, w_agr=0,
                  nlm_margin=2, nlm_k=6, beam_s=8, nlm_norm="self")
print(xl.convert("mai ghr pe hu"))            # मैं घर पे हूं

The Python package is the reference used for training and evaluation; the JavaScript runtime reproduces its decisions exactly (verified sentence by sentence on the casual set and on the evaluation sets: identical v2 and v3 outputs, identical accuracies).

API at a glance

Call	Returns
attachSentence(engine, nlmBytes)	attaches sentence mode; returns {nlm, info} where info has the decoding params, the parameter count (2,770,752) and the LM config
engine.sentenceMode	true (sentence) or false (word mode) — toggles the decoder without reloading anything
engine.transliterate(text, {mode, context, k, overrides})	same result object as the word model, plus guard: {ok, words, outside}
search3(lattice, params, coreLm, nlm)	the beam search itself, if you build your own lattice (format documented at the top of v3.mjs)

Files

File	What it is	Size (gz)
nlm.q8	the sentence LM — 2,770,752 params, 8-bit, tokenizer embedded	2.92 MB
v3.mjs	lattice beam search (search3)	2.1 KB
nlm.mjs	LM file reader + inference	3.1 KB
sent.mjs	glue: builds the lattice from the word model, toggles word/sentence mode, audits the guardrail	1.8 KB
xlit.q4, params.json, dict/	the v2 word model this builds on (unchanged)	7.92 MB total
aksharalp.mjs, aksharalp.min.mjs	the word-model runtime	20 KB
manifest.json	file list, sizes, checksums	—
python/	reference Python implementation (numpy + PyTorch)	—
cases/	the 21-sentence casual set used for the tables above	—
assets/	banner, charts, screenshots (assets/screenshots/)	—
LICENSE, NOTICE	Apache-2.0, third-party attributions	—

Decode settings (all fixed before the test run): w_nlm 0.4, w_lm 0.25, w_ch_s 1.5, w_agn 2, w_agr 0, nlm_margin 2, nlm_k 6, beam_s 8, lat_k 12, rec_n 3, nlm_norm self.

Limitations

• Hindi only. One script pair: romanized Hindi → Devanagari. No other Indic language, no reverse direction.
• It cannot add a candidate it was not given. If the word model's top-6 misses the right spelling (see Boht → बोहत), no amount of sentence context finds it. That is a channel/recall problem and it is the single biggest remaining error source.
• Two-word inputs gain nothing. Short fragments with no verb or auxiliary have no sentence signal (nahi pata → नहीं पाता). Add the subject and it works (mujhe nahi pata → मुझे नहीं पता).
• Chat normalisation cuts both ways. The LM sometimes rewrites deliberate chat spellings into dictionary forms (extra h, doubled letters, stretched vowels): see the regressions table.
• Casual-chat gains are not significant. On our 480-sentence casual set the sentence-accuracy gain is +1.46 points with a 95% CI of [−0.42, 3.54]. The significant wins are on the held-out Dakshina sentence test (sentence accuracy and word accuracy) and on the context set in word accuracy (+2.13 points, CI [0.42, 4.64]); the context set has only 66 sentences, so its sentence-level +4.55 has a CI of [0.00, 10.61] and is not flagged.
• Names, rare words and code-mixing remain the weak spot, the same as the word model: proper nouns, brand-new slang, heavy English mixing.
• Speed is CPU JavaScript. 95 ms per sentence at the shipped setting on a shared box; a long sentence, a cold cache or an old phone is slower. It is not a keystroke-latency budget at full sentence length — the demo debounces.
• One decoding setting. These weights were tuned for nlm_norm: "self". Changing to exact normalisation needs a re-tune to recover the 0.1–0.3 points it is worth on dev data.
• Not a translator. It changes the script, not the language.

How it fits the family

Model	What it is	Size	Best at
AksharaLP-1M	the first tiny word model, core only	4.53 MB	smallest footprint, standalone word conversion
AksharaLP-1M-v2	the word model + dictionary pack this repo builds on	7.92 MB	word-mode accuracy — 70.04 top-1 / 88.31 top-3 / 93.63 top-8 on Aksharantar Hindi test; best top-3 and top-8 in the family and best on chat spellings (v1 edges it on top-1 by 0.33 pts, CIs overlap)
AksharaLP-Kosh-1.2M	the sub-megabyte word model (its own card is CC-BY-SA-4.0)	0.78 MB	the smallest possible download at 68.09% top-1
AksharaLP-Sentence-2.8M (this)	v2 word model + a 2.77M neural sentence LM	10.85 MB	sentence accuracy — 26.73% sentences exact and 89.44% words on the Dakshina sentence test, the best in the family

Kosh is the word model for when one megabyte is the whole budget — 0.78 MB, 68.09% top-1, with its own licence. This is the sentence-accurate one: same word-mode candidates, plus a language model that reads the sentence. v2 remains the best word-mode accuracy in the family on top-3/top-8 and on casual chat (AksharaLP-1M is 0.33 points ahead on top-1 alone, inside overlapping intervals) — this repo does not claim otherwise, and does not change word mode at all.

Licence & attribution

Commercial use is allowed. Code and model weights ship under the Apache License 2.0 (see LICENSE), Copyright 2026 Sourojit Dhua.

The sentence LM is commercially clean, and that is the important part. It was trained only on IndicCorp v2 Hindi (CC0) with our own tokenizer, our own training code and our own runtime. No non-commercial corpus, no scraped chat data, no third-party model weights. The full system still inherits the v2 word package's terms (Apache-2.0, with the data attributions below listed in NOTICE).

Attribution is required when you redistribute it. Apache-2.0 §4(d): include NOTICE in every redistribution or derivative, and keep the runtime's /*! … */ header (including in aksharalp.min.mjs).

Transliteration by AksharaLP (Sourojit Dhua) — https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M

Source	Licence	Used for
IndicCorp v2, AI4Bharat	CC0	sentence LM training data; word-model lexicon and bigram LM
Aksharantar, AI4Bharat	CC0 (mined) / CC-BY (manually collected)	word-model training pairs
IndicXlit, AI4Bharat	MIT	teacher model for word-model distillation
CMUdict, Carnegie Mellon University	BSD-2-Clause-style	English word cues in the word model

Fonts in the demo Space (Inter, Noto Sans Devanagari) are under the SIL Open Font License 1.1.

Citation

@software{dhua_aksharalp_sentence_2026,
  author    = {Dhua, Sourojit},
  title     = {{AksharaLP-Sentence}: Tiny, Sentence-Level Hindi Transliteration for the Browser},
  year      = {2026},
  version   = {3.0.0},
  url       = {https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M},
  note      = {Word model: AksharaLP-1M-v2 (989K parameters). Sentence LM: 2.77M parameters, trained on IndicCorp v2 Hindi (CC0).},
  license   = {Apache-2.0}
}

_{AksharaLP · sentence model v3.0.0 · built 2026-11-04 · Questions and feedback: open a discussion on this repository.}