README.md · sourojitdotcom/AksharaLP-1M at main

Improve README: versions, metadata, wording
5d04d17 verified 4 days ago
31.4 kB
---
license: apache-2.0
new_version: sourojitdotcom/AksharaLP-1M-v2
language:
  - hi
  - en
pipeline_tag: translation
tags:
  - transliteration
  - hindi
  - devanagari
  - romanized-hindi
  - on-device
  - browser
  - javascript
  - tiny
  - indic
  - input-method
  - keyboard
  - offline
  - aksharalp
datasets:
  - ai4bharat/Aksharantar
  - ai4bharat/IndicCorpV2
metrics:
  - accuracy
  - cer
model-index:
  - name: AksharaLP-1M
    results:
      - task:
          type: translation
          name: Romanized Hindi → Devanagari transliteration (word level)
        dataset:
          type: ai4bharat/Aksharantar
          name: Aksharantar (Hindi test)
          split: test
        metrics:
          - type: accuracy
            name: Top-1 accuracy
            value: 70.37
          - type: accuracy
            name: Top-3 accuracy
            value: 87.66
          - type: accuracy
            name: Top-8 accuracy
            value: 92.79
      - task:
          type: translation
          name: Romanized Hindi → Devanagari transliteration (sentence level)
        dataset:
          type: google/dakshina
          name: Dakshina (Hindi romanized sentences)
          split: test
        metrics:
          - type: accuracy
            name: Word accuracy
            value: 87.55
          - type: exact_match
            name: Sentence exact match
            value: 21.66
          - type: cer
            name: CER
            value: 4.81
---
<p align="center">
  <img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/banner.png" alt="AksharaLP — Hindi transliteration in 4.53 MB" width="100%"/>
</p>

<p align="center">
  <a href="https://huggingface.co/spaces/sourojitdotcom/AksharaLP-demo"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Live%20demo-Space-6d4aff" alt="Live demo"/></a>
  <img src="https://img.shields.io/badge/core-0.57%20MB-6366f1" alt="Core download size"/>
  <img src="https://img.shields.io/badge/with%20dictionary-4.53%20MB-a855f7" alt="Complete download size"/>
  <img src="https://img.shields.io/badge/parameters-989K-ec4899" alt="Parameters"/>
  <img src="https://img.shields.io/badge/runs-100%25%20on--device-10b981" alt="On-device"/>
  <img src="https://img.shields.io/badge/license-Apache--2.0-f97316" alt="License"/>
</p>

# AksharaLP-1M

> [!IMPORTANT]
> **This is v1 (1.0.0). [AksharaLP-1M-v2](https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2) is the recommended version** ([v2 live demo](https://huggingface.co/spaces/sourojitdotcom/AksharaLP-v2-demo)).
> Same 989K-parameter core architecture; v2 ships a 3M-word dictionary pack (v1: 1.5M) and a reranker refitted for chat-style spellings (*h*, *nhi*, *bht*).
>
> | | v1 (this repo) | v2 |
> |---|---|---|
> | Complete download (gzipped) | **4.53 MB** | 7.92 MB |
> | Aksharantar top-1 / top-3 / top-8 % | **70.37** / 87.66 / 92.79 | 70.04 / **88.31** / **93.63** |
> | Core-only top-1 % | 53.36 | **56.01** |
> | Dakshina word acc / sentence EM % | 87.55 / 21.66 | **89.02** / **25.70** |
> | Chat-spelling sentences, word acc % (480 held-out) | 80.16 | **91.54** |
> | Browser latency / word (p50) | 219.7 ms | **206.0 ms** |
>
> <sub>Numbers from the two model cards; the chat-spelling comparison is from the v2 card. Top-1 on Aksharantar differs by 0.33 points with overlapping 95% CIs ([69.50, 71.26] vs [69.16, 70.90]).</sub>
> Keep using v1 only if the smaller 4.53 MB download matters more than chat-spelling accuracy. v1 stays available for reproducibility.

<h3 align="center">The world’s 3rd most spoken language, typed on the world’s most used keyboard. In 4.5 MB.</h3>
<p align="center"><b>Hindi typed in English letters → Devanagari, fully offline.</b></p>

### Why it matters

| 🌏 **#3** | 🗣️ **611M** | 📦 **4.5 MB** |
|:---:|:---:|:---:|
| most spoken language in the world | people speak Hindi (first or second language) | is all it takes to type it — no server, no GPU |

<sub>Source: Ethnologue 200 (Ethnologue: Languages of the World, 29th edition, SIL Global), 2026 — <a href="https://www.ethnologue.com/insights/ethnologue200/">https://www.ethnologue.com/insights/ethnologue200/</a> (accessed 2026-10-31). Counts first- and second-language speakers.</sub>

On a Latin (QWERTY) keyboard, Hindi is often typed in English letters: *“kal meeting hai, kaise ho?”*. AksharaLP-1M turns that into
proper Devanagari — accurately, in context, and entirely offline in the browser. It transliterates (changes the script); it does not translate.

AksharaLP-1M is the 989K-parameter member of the **AksharaLP** family of tiny transliteration models.

For *“kal meeting hai, kaise ho?”* AksharaLP writes **कल मीटिंग है, कैसे हो?**, the way a Hindi speaker would.
It ships in two tiers: a **0.57 MB core** (character Transformer with 989K parameters, reranker and runtime) that works on its own,
and a **3.97 MB dictionary pack** (1.5M-word lexicon and sentence language model) that the runtime fetches on first use and caches.
The complete package is **4.53 MB** to download.
The complete package is **5× smaller** than the model weights of IndicXlit, the most widely used open Indic transliteration model,
and **47× smaller** than IndicXlit with its rescoring dictionary — while scoring **higher top-1 than both** on Aksharantar (70.37% vs 59.16% for the model and 69.33% for the full package; the gap to the full package is within overlapping 95% CIs).

<p align="center">
  <a href="https://huggingface.co/spaces/sourojitdotcom/AksharaLP-demo"><img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/typing.gif" alt="Live typing demo" width="88%"/></a>
  <br/><sub>Recorded from the <a href="https://huggingface.co/spaces/sourojitdotcom/AksharaLP-demo">live demo</a>: inference runs in a Web Worker in the browser tab.</sub>
</p>

## Highlights

|  |  |
|---|---|
| 🪶 **Extremely small** | 0.57 MB core + 3.97 MB dictionary pack = 4.53 MB gzipped (8.69 MB on disk); 989K parameters, 4-bit weights. Fits in a keyboard app, a browser extension or a web page. |
| 🎯 **Accurate** | With the dictionary pack: 70.37% top-1 / 87.66% top-3 on the Aksharantar Hindi test set (9,991 words); 87.55% word accuracy on Dakshina sentences. Core-only numbers are in [Variants](#variants-core-only-vs-core--dictionary). |
| 🧠 **Context-aware** | A compact bigram language model picks the right word from its neighbours: the same *main* becomes मैं or में. |
| 📥 **Starts fast, degrades gracefully** | The core is enough to start typing; the dictionary pack streams in after and is cached. If the pack cannot be fetched, the runtime keeps working with the core alone. |
| 🔒 **Private & offline** | Pure JavaScript, zero dependencies, no server. Nothing you type leaves the device; works offline after first load. |
| ⚡ **Fast** | 219.7 ms per word (p50) in the browser on an 8-vCPU Intel Xeon cloud VM (headless Chrome 156, Linux). |

## What it brings on-device

Everything below happens on the user’s own phone or laptop: the model is a set of static files plus a dependency-free JavaScript runtime.
Each example output was produced by the released runtime when this card was built.

<table>
<tr>
<td width="33%" valign="top">

### ✈️ Fully offline
No network after the first load, so it keeps working in airplane mode, in a lift, or on a train.

`flight mein hoon, land karke call karta hoon`<br/>→ **फ्लाइट में हूं, लैंड करके कॉल करता हूं**

</td>
<td width="33%" valign="top">

### 🔒 Private by design
Text never leaves the device: no API call, no logging server. Suited to banking, health and personal chat.

`mera account balance kitna hai`<br/>→ **मेरा अकाउंट बैलेंस कितना है**<br/>
`doctor ne din mein do baar dawai lene ko kaha`<br/>→ **डॉक्टर ने दिन में दो बार दवाई लेने को कहा**

</td>
<td width="33%" valign="top">

### 🌐 Runs wherever JavaScript runs
A browser tab, an Android or iOS WebView, an installable PWA, or Node.js. CPU only: no GPU, no server, no inference bill.

`namaste, aapka swagat hai`<br/>→ **नमस्ते, आपका स्वागत है**

</td>
</tr>
<tr>
<td width="33%" valign="top">

### ⚡ Real-time, as you type
The sentence is re-decoded on every keystroke in a Web Worker, and earlier words are corrected as context arrives:

`main` → मैं<br/>
`main theek` → मैं ठीक<br/>
`main theek hoon` → **मैं ठीक हूं**

<sub>Browser latency: 219.7 ms per word (p50) on an 8-vCPU Intel Xeon cloud VM (headless Chrome 156, Linux).</sub>

</td>
<td width="33%" valign="top">

### ⌨️ Keyboard & input-box integration
Wire it to any `<input>` or `<textarea>`: chat, search, forms, CRM notes, customer-support replies. One call per keystroke: `transliterate(text, {k})`.

`aapka refund teen din mein aa jayega`<br/>→ **आपका रिफंड तीन दिन में आ जाएगा**

</td>
<td width="33%" valign="top">

### 🏷️ Brands your way
English words inside Hindi sentences are transliterated by the same model. Two writing styles for the same input:

Default → **कल व्हाट्सएप पे मीटिंग है**<br/>
Strict (brands stay in Latin) → **कल whatsapp पे मीटिंग है**

</td>
</tr>
<tr>
<td width="33%" valign="top">

### ☰ Top-k suggestions
Ranked candidates for every word, ready for a suggestion bar or candidate picker. Picking one re-decodes the rest of the sentence.

`main` → <b>मैं</b> · मैन · मेन · माइन · में

</td>
<td width="33%" valign="top">

### 📶 Built for Bharat
A 0.57 MB core to start, 4.53 MB in total, downloaded once and cached; after that, patchy or missing connectivity does not matter. Pure JavaScript on the CPU, with no GPU or native code, designed with entry-level Android phones in mind.

`signal nahi aa raha, baad mein baat karte hain`<br/>→ **सिग्नल नही आ रहा, बाद में बात करते हैं**

</td>
<td width="33%" valign="top">

### ₹0 per call at scale
Inference runs on your users’ devices, so there are no per-request API fees and no GPU servers to scale. Serving is static files from any CDN. Apache-2.0: free for commercial use.

`mera order kab tak deliver hoga`<br/>→ **मेरा ऑर्डर कब तक डिलीवर होगा**

</td>
</tr>
</table>

## Best in class for its size

<table>
<tr><td align="center" valign="top" width="33%"><h2>70.4%</h2><b>top-1 — higher than IndicXlit (model only) (59.2%)</b><br/><sub>at 5× smaller · Aksharantar Hindi test</sub></td><td align="center" valign="top" width="33%"><h2>70.4%</h2><b>top-1 — higher than IndicXlit (full package) (69.3%)</b><br/><sub>at 47× smaller · Aksharantar Hindi test</sub></td><td align="center" valign="top" width="33%"><h2>8.1 pts/MB</h2><b>highest top-1 accuracy per MB among compared models</b><br/><sub>next: IndicXlit (model only) at 1.28 pts/MB</sub></td></tr>
<tr><td align="center" valign="top" width="33%"><h2>87.7%</h2><b>right word in the first 3 suggestions — vs 59.2% top-1 for IndicXlit (model only)</b><br/><sub>Aksharantar Hindi test</sub></td><td align="center" valign="top" width="33%"><h2>87.7%</h2><b>right word in the first 3 suggestions — vs 69.3% top-1 for IndicXlit (full package)</b><br/><sub>Aksharantar Hindi test</sub></td><td align="center" valign="top" width="33%"><h2>5×</h2><b>smaller than IndicXlit (model only)</b><br/><sub>8.69 MB vs 46.0 MB on disk (fp32 weights)</sub></td></tr>
<tr><td align="center" valign="top" width="33%"><h2>47×</h2><b>smaller than IndicXlit (full package)</b><br/><sub>8.69 MB vs 404 MB on disk (fp32 weights + 358 MB rescoring dictionary)</sub></td><td align="center" valign="top" width="33%"><h2>219.7 ms</h2><b>per word (p50) in the browser</b><br/><sub>an 8-vCPU Intel Xeon cloud VM (headless Chrome 156, Linux)</sub></td><td align="center" valign="top" width="33%"><h2>4.53 MB</h2><b>complete download</b><br/><sub>core + 1.5M-word dictionary pack · 989K parameters</sub></td></tr>
<tr><td align="center" valign="top" width="33%"><h2>0.57 MB</h2><b>to start typing</b><br/><sub>the core works on its own (53.4% top-1 without the pack); the dictionary pack streams in after and is cached</sub></td></tr>
</table>

**Accuracy per MB** (top-1 on Aksharantar Hindi test ÷ on-disk size; systems measured by us with the same harness):

| # | System | On disk | Top-1 % | Top-1 pts per MB |
|---|---|---|---|---|
| 1 | **AksharaLP-1M** | 8.69 MB | 70.37 | **8.10** |
| 2 | IndicXlit (model only) | 46.0 MB | 59.16 | 1.28 |
| 3 | IndicXlit (full package) | 404 MB | 69.33 | 0.17 |

> 🏆 No compared model is both smaller and more accurate: AksharaLP-1M sits on the size–accuracy frontier.

<sub>Every tile above is generated by the release build from measured results and shown only if its rule holds (rules and outcomes are logged in the build report).</sub>


## What it does — real outputs

Every output below was produced by the released runtime (`aksharalp.mjs`) and the files in this repository when this card was built.

| Feature | You type | AksharaLP writes |
|---|---|---|
| Everyday Hindi | `kaise ho` | **कैसे हो** |
| Everyday Hindi | `mujhe bhookh lagi hai` | **मुझे भूख लगी है** |
| Context — *main* = “I” | `main theek hoon` | **मैं ठीक हूं** |
| Context — *main* = “in” | `ghar main kaun hai` | **घर में कौन है** |
| English words in Hindi | `kal office mein meeting hai` | **कल ऑफिस में मीटिंग है** |
| English words in Hindi | `whatsapp pe message karo` | **व्हाट्सएप पे मेसेज करो** |
| Acronyms | `sbi` | **एसबीआई** |
| Acronyms | `SBI ka ATM kahan hai` | **एसबीआई का एटीएम कहां है** |
| English words | `please mujhe call karna` | **प्लीज मुझे कॉल करना** |
| Kept as-is | `@rahul 5 baje milte hain, link: www.example.com` | **@rahul 5 बजे मिलते हैं, लिंक: www.example.com** |

### Two writing styles

The same sentence in each mode:

| Mode | `kal whatsapp pe meeting ka link bhej dena` |
|---|---|
| Default | कल व्हाट्सएप पे मीटिंग का लिंक भेज देना |
| Strict | कल whatsapp पे मीटिंग का लिंक भेज देना |

* **Default** — every word, English ones included, is written in Devanagari.
* **Strict** — brand and product names stay in Latin script.

### Top-k suggestions

Every word comes with ranked alternatives, ready for a keyboard suggestion bar:

| Input | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| `kaise` | **कैसे** | केसे | कइसे | कैस | किसे |
| `main` | **मैं** | मैन | मेन | माइन | में |
| `dhanyavaad` | **धन्यवाद** | धन्यावाद | धन्यवाद् | धान्यवाद | धन्यवादी |

### Core only vs. core + dictionary pack

The same inputs through the core alone (pack not loaded) and with the pack, from the released runtime:

| You type | Core only | Core + dictionary pack |
|---|---|---|
| `main theek hoon` | मैं थीक हूं | **मैं ठीक हूं** |
| `ghar main kaun hai` | घर मैं कौं है | **घर में कौन है** |
| `mujhe bhookh lagi hai` | मुझे भूख लगी है | **मुझे भूख लगी है** |
| `whatsapp pe message karo` | व्हाटसएप पीई मेसेज करो | **व्हाट्सएप पे मेसेज करो** |
| `doctor ne dawai lene ko kaha` | डॉक्टर ने दवाई लेने को कह | **डॉक्टर ने दवाई लेने को कहा** |

## Small by design

<p align="center"><img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/chart_size.png" alt="Size comparison" width="92%"/></p>

What you download:

| File | What it is | On disk | gzipped |
|---|---|---|---|
| `xlit.q4` | core: transliteration Transformer, 4-bit weights | 589 KB | 544 KB |
| `params.json` | core: reranker weights + decoding settings | 12 KB | 5 KB |
| `aksharalp.mjs` | core: JavaScript runtime (zero dependencies) | 52 KB | 18 KB |
| **Core total** | works on its own | **0.65 MB** | **0.57 MB** |
| `dict/lex.bin` | dictionary pack: Hindi lexicon (trie + unigram scores) | 5,647 KB | 3,137 KB |
| `dict/lm.bin` | dictionary pack: word-bigram language model, 4-bit | 2,373 KB | 826 KB |
| `dict/params.json` | dictionary pack: reranker weights + decoding settings used with the pack | 12 KB | 5 KB |
| **Dictionary pack total** | fetched on first use, then cached | **8.03 MB** | **3.97 MB** |
| **Complete package** | core + pack | **8.69 MB** | **4.53 MB** |

For comparison, IndicXlit’s fp32 model weights are 46.0 MB and its official rescoring dictionary adds another 358 MB.
General-purpose LLMs that can transliterate are larger still, with hundreds of millions of parameters or more, and typically need a server or a GPU.

## Benchmarks

<p align="center"><img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/chart_accuracy.png" alt="Accuracy comparison" width="92%"/></p>

| System | Download | Aksharantar top-1 % | top-3 % | top-8 % | Dakshina word acc % | Dakshina sentence EM % | Latency / word (p50) |
|---|---|---|---|---|---|---|---|
| **AksharaLP-1M** (core + dictionary pack) | **4.53 MB** | **70.37** <sub>[69.50, 71.26]</sub> | **87.66** | **92.79** | **87.55** | **21.66** | **219.70 ms** <sub>browser</sub> |
| IndicXlit (model only) | 46.0 MB <sub>fp32 weights</sub> | 59.16 <sub>[58.14, 60.13]</sub> | 79.95 | 88.29 | 67.55 | 6.29 | 48.4 ms <sub>PyTorch CPU</sub> |
| IndicXlit (full package) | 404 MB <sub>fp32 weights + 358 MB rescoring dictionary</sub> | 69.33 <sub>[68.44, 70.23]</sub> | 85.40 | 88.29 | — | — | — |

<p align="center"><img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/chart_tradeoff.png" alt="Accuracy versus size" width="80%"/></p>

## Variants: core only vs. core + dictionary

The dictionary pack is optional. Each row below was scored once on the Aksharantar Hindi test set (9,991 words, beam 8); settings and the size of the
smaller dictionary were chosen on the validation set only. The 200K-word pack is measured for comparison but **not shipped** in this release.

| Variant | Shipped | Download (gz) | On disk | Test top-1 % <sub>[95% CI]</sub> | top-3 % | top-8 % | Val top-1 % <sub>(2-fold CV)</sub> | Test top-1 % with test words removed from the dictionary | Dakshina sentences word acc % |
|---|---|---|---|---|---|---|---|---|---|
| **Core only** | yes | 0.57 MB | 0.65 MB | 53.36 <sub>[52.36, 54.32]</sub> | 74.52 | 83.81 | 53.92 | — | 65.35 |
| Core + 200K-word dictionary | no (measured only) | 1.13 MB | 1.76 MB | 68.38 <sub>[67.48, 69.30]</sub> | 85.70 | 91.58 | 67.35 | 33.33 | 86.89 |
| **Core + 1.5M-word dictionary pack** | yes | 4.53 MB | 8.69 MB | 70.37 <sub>[69.50, 71.26]</sub> | 87.66 | 92.79 | 69.30 | 19.65 | 87.55 |

<sub>The “test words removed” column re-runs the same system with every Aksharantar test reference word deleted from the dictionary, so the dictionary cannot
simply look up the answer; it shows how much of the gain comes from dictionary coverage (see the caveat below). Dakshina sentences: 4,976 romanized sentences.</sub>

**How we measured.** All systems were run by us on the same machine with the same scoring script (Unicode NFC, punctuation and danda normalised).
Word-level: Aksharantar Hindi test set, 9,991 unique romanized words, beam 8 for every system; top-k = the reference appears among the first k candidates.
Sentence-level: Dakshina romanized Hindi sentences (4,976 sentences); word accuracy, sentence exact match, CER and WER.
IndicXlit “model only” is the released transformer checkpoint; “full package” adds rescoring with the official 4.2M-word dictionary (α = 0.9; our re-implementation of the released engine, which reproduces the published scores).
Browser latency is per word with the dictionary pack loaded, single tab, measured in headless Chrome on an 8-vCPU Intel Xeon cloud VM (headless Chrome 156, Linux); IndicXlit latency is PyTorch fp32 on one CPU thread and is shown for reference only (different runtime).

> **Caveat.** IndicXlit was trained on Aksharantar, so its Aksharantar test scores may benefit from train/test overlap. Our training pairs contain no Aksharantar test or validation native words (checked). The 1.5M-word dictionary pack is built from IndicCorp v2 text and naturally contains most test words: with every test reference word removed from the dictionary the same system scores 19.65% top-1 (47.27% top-3, 73.06% top-8), and the core alone scores 53.36% top-1. All tuning, the reranker fit and the choice of dictionary size used the validation set only; each test number comes from a single run.

## Quickstart

### Browser (ES module)

```html
<script type="module">
  import { AksharaLP } from "https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/aksharalp.mjs";
  const base = "https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/";
  // core first (0.57 MB): usable immediately
  const aklp = await AksharaLP.load(base, { dict: false });
  // then the dictionary pack (3.97 MB): fetched once, kept in the Cache API; false = stays core-only
  const ok = await aklp.loadDict();
  const r = aklp.transliterate("kal meeting hai, kaise ho?");
  console.log(r.output);                       // कल मीटिंग है, कैसे हो? (with the pack)
  console.log(r.tokens[0].candidates);         // ranked alternatives for the first word
</script>
```

`AksharaLP.load(base)` without options loads the core and the pack together. For production, copy the files (including `dict/`) into your app and load them from your own origin (they are static and cache well); run the model in a Web Worker as the demo Space does.

### Node.js (≥ 18)

```bash
hf download sourojitdotcom/AksharaLP-1M --local-dir aksharalp
```

```js
import { loadFromDir } from "./aksharalp/aksharalp.mjs";
const aklp = await loadFromDir("./aksharalp/");            // core + dict/ pack; { dict: false } for core only
console.log(aklp.transliterate("main theek hoon", { mode: "conversational" }).output);   // मैं ठीक हूं
console.log(aklp.word("kaise", 5));                                                      // top-5 candidates with scores
```

### Python (reference implementation)

```python
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("sourojitdotcom/AksharaLP-1M")
sys.path.insert(0, f"{path}/python")
from bhc.translit2 import BhcTranslit2          # needs numpy + torch (CPU)
xl = BhcTranslit2(path)                         # use_dict=False for core only
print(xl.convert("kal meeting hai, kaise ho?"))
print(xl.translit_words(["kaise", "namaste"], k=5))
```

The Python package is the reference used for training and evaluation; the JavaScript runtime reproduces it exactly
(JS/Python parity test — core only: identical candidate sets and top-1 for 196/196 words, max feature difference 9.1e-06; with the dictionary pack: identical candidate sets and top-1 for 196/196 words, max feature difference 3.2e-05).

### API at a glance

| Call | Returns |
|---|---|
| `AksharaLP.load(baseUrl, {dict, cache})` | engine (fetches `manifest.json` and the core; with `dict: true`, the default, also the dictionary pack) |
| `loadDict({cache})` | fetches the dictionary pack (Cache API name `"aksharalp-dict"` by default) and attaches it; returns `false` and keeps the core working if it fails |
| `attachDict({lex, lm, params})` | attaches a pack you fetched yourself (e.g. from IndexedDB or app assets) |
| `transliterate(text, {mode, context, k, overrides})` | `{output, tokens[], ms}`; each token has `text, output, source, candidates[]`; `mode` is `"conversational"` (default) or `"strict"` |
| `word(romanWord, k)` | top-k `{text, score}` for one word |
| `info()` | parameter count, lexicon size, bytes on disk / gzipped, `hasDict`, `dictError` |

`overrides` pins a user-chosen suggestion (`{tokenStartOffset: "मैं"}`) and re-decodes the rest of the sentence around it.

## How it works

<p align="center"><img src="https://huggingface.co/sourojitdotcom/AksharaLP-1M/resolve/main/assets/architecture.png" alt="Architecture" width="100%"/></p>

```mermaid
flowchart LR
    A["Romanized Hindi text"] --> T["① Tokenizer<br/>URLs, @mentions, numbers kept"]
    T -->|Strict mode: brand| P["kept in Latin"]
    T -->|word| X["② Char Transformer (core)<br/>d=128, 3+2 layers, 4 heads, FFN 384 · 4-bit · 989K"]
    X --> C["Candidates<br/>free beam + acronyms<br/>(+ lexicon-trie beam + skeleton lookups with the pack)"]
    C --> R["Learned reranker<br/>24-feature MLP"]
    R --> V["③ Sentence Viterbi (pack)<br/>bigram LM · 1.5M words"]
    P --> V
    V --> O["Devanagari + top-k"]
```

1. **Tokenizer.** Splits words, numbers and punctuation; URLs, e-mail addresses and @mentions are kept as typed.
2. **Transliteration (core).** A character-level encoder–decoder Transformer (d=128, 3+2 layers, 4 heads, FFN 384) proposes spellings with a free beam; letter-name acronyms are added and scored by the same model. With the dictionary pack, a second beam constrained to the 1.5M-word Hindi lexicon trie and consonant-skeleton lookups add in-vocabulary candidates. A small learned reranker (24 features → 16 hidden units → 1, fitted on the validation set) orders the candidates; the core and the pack each carry their own reranker weights.
3. **Sentence context (pack).** A 4-bit quantized word-bigram language model (300K bigrams) runs Viterbi over the candidate lattice to choose the most fluent sentence. Core-only mode picks each word independently.

English words inside Hindi sentences go through the same model (Default) or, for brand names, stay in Latin script (Strict).

### Files

| File | Contents |
|---|---|
| `xlit.q4` | core: transliteration Transformer, 4-bit weights |
| `params.json` | core: reranker weights + decoding settings |
| `dict/lex.bin` | dictionary pack: Hindi lexicon (trie + unigram scores) |
| `dict/lm.bin` | dictionary pack: word-bigram language model, 4-bit |
| `dict/params.json` | dictionary pack: reranker weights + decoding settings used with the pack |
| `manifest.json` | file list, sizes, checksums (core + pack) |
| `aksharalp.mjs` | core: JavaScript runtime (zero dependencies) |
| `aksharalp.min.mjs` | same runtime, minified (attribution header preserved) |
| `python/` | reference Python implementation (numpy + PyTorch) |
| `LICENSE`, `NOTICE` | Apache-2.0 and third-party attributions |

## Intended use

* Hindi input methods: mobile and desktop keyboards, browser extensions, chat and messaging apps, search boxes.
* Converting romanized Hindi text (chat logs, social posts, transcripts) to Devanagari for downstream NLP.
* Privacy-sensitive or offline settings where text must not leave the device.

## Limitations

* **Hindi only.** Other Indic languages and scripts are not supported.
* **Names and rare words.** Proper nouns, rare or very long words, and unusual spellings are the main source of errors. Check the top-k list.
* **Ambiguity.** Some romanizations have several valid Devanagari spellings (e.g. हूं / हूँ). The model follows the most common written form, which may differ from a reference.
* **Short context.** The language model sees one word on each side (bigram), so long-range agreement is not modelled.
* **Core only is less accurate.** Without the dictionary pack, top-1 accuracy drops substantially (see [Variants](#variants-core-only-vs-core--dictionary)). Load the pack when you can.
* **Dictionary coverage.** Much of the accuracy with the pack comes from the lexicon: with every test reference word removed from it, top-1 falls well below the core-only score. Words outside the lexicon (new names, slang, rare spellings) are harder.
* **English words and language tags.** This release has no word-level language identifier and no loanword lexicon: English words are transliterated phonetically by the same model and can look unusual, and tokens are not tagged Hindi / English.
* **Short ambiguous words.** Without a loanword lexicon, short words that are both English and Hindi can go the wrong way in context (for example, in our tests *call* sometimes came out as कल and *do* as डॉ). Check the top-k list.
* **Chat-speak.** Heavy abbreviation (e.g. *kr, rhe, btw*) is handled only partly.
* **Speed** depends on the device. Very old phones may see higher per-word latency than reported here.

## License & attribution

**Commercial use is allowed.** Code and model weights are released under the **Apache License 2.0** (see `LICENSE`), Copyright 2026 Sourojit Dhua.
You may use, modify and ship AksharaLP-1M in commercial products, free of charge.

**Attribution is required when you redistribute it.** Apache-2.0 §4(d) requires every redistribution of the model or a derivative
(shipping the files in an app, web page, browser extension or SDK; publishing a fine-tuned, quantized or converted version)
to include the `NOTICE` file, and the JavaScript runtime keeps its `/*! … */` attribution header, including the minified `aksharalp.min.mjs`.
Please use this attribution text:

```text
Transliteration by AksharaLP (Sourojit Dhua) — https://huggingface.co/sourojitdotcom/AksharaLP-1M
```

**Recommended in-app credit** (About, Settings or Licenses screen):

```html
Hindi typing powered by <a href="https://huggingface.co/sourojitdotcom/AksharaLP-1M">AksharaLP by Sourojit Dhua</a>
```

Third-party data used to build the model is credited in `NOTICE`:

| Source | License | Used for |
|---|---|---|
| [Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar) (AI4Bharat) | CC0 (mined data) / CC-BY (manually collected data) | romanized ↔ Devanagari training pairs |
| [IndicXlit](https://github.com/AI4Bharat/IndicXlit) (AI4Bharat) | MIT | teacher model for distillation |
| [IndicCorp v2](https://github.com/AI4Bharat/IndicBERT/tree/main#indiccorp-v2) (AI4Bharat) | CC0 | Hindi lexicon and bigram language model |
| [CMUdict](https://github.com/cmusphinx/cmudict) (Carnegie Mellon University) | BSD-2-Clause-style | English word cues |


Fonts in the demo Space (Inter, Noto Sans Devanagari) are under the SIL Open Font License 1.1.

## Citation

```bibtex
@software{dhua_aksharalp_2026,
  author    = {Dhua, Sourojit},
  title     = {{AksharaLP}: Tiny, Context-Aware Hindi Transliteration for the Browser},
  year      = {2026},
  version   = {1.0.0},
  url       = {https://huggingface.co/sourojitdotcom/AksharaLP-1M},
  license   = {Apache-2.0}
}
```

<sub>AksharaLP · v1.0.0 · built 2026-10-31 · Questions and feedback: open a discussion on this repository.</sub>