1.2.0: fix words cut short (acha, achha -> अच्छा); Hinglish set 112 -> 113, Aksharantar test unchanged

README.md

@@ -62,9 +62,9 @@ console.log(tr.word("namaste").candidates.slice(0, 4).map((c) => c.text));
 |---|---|---|
 | `model.q4.gz` | the model: encoder-decoder transformer, 2 + 2 layers, width 128, 811,904 parameters at 4 bits | 432,512 bytes |
 | `words.bin.gz` | 200,000 common Hindi words in 8 frequency bands | 432,193 bytes |
-| `translit.js` | the runner (29.5 KB as written) | 10,308 bytes |
+| `translit.js` | the runner (30.9 KB as written) | 10,900 bytes |
 | `model.json.gz`, `rerank.json` | settings and alphabet | 1,248 bytes |
-| **Total download** | | **876,261 bytes (0.88 MB)** |
+| **Total download** | | **876,853 bytes (0.88 MB)** |
 
 Speed: about 2 ms a word with WebAssembly SIMD, which current browsers have (1.9 ms a word in Node on a Mac mini over all
 9,991 Aksharantar test words; 2.4 to 2.8 ms on everyday words in Chrome and Firefox on an Intel laptop). Without SIMD it
@@ -78,7 +78,7 @@ paired for gaps. Measured by us on 28 October 2026.
 
 | System | What you download or hold | Aksharantar Hindi | Hinglish chat set (of 122) |
 |---|---|---|---|
-| **Lekhani** | **0.88 MB** | **65.9%** (64.9 to 66.8) | **112** |
+| **Lekhani** | **0.88 MB** | **65.9%** (64.9 to 66.8) | **113** |
 | IndicXlit, model on its own (beam 4, its default) | 11.5M parameters (46 MB fp32; a third-party 8-bit browser export is 14.4 MB) | 58.9% | 102 |
 | IndicXlit, full Python package (model + word list) | the model + a 4.2M-word list, 358 MB in memory | 68.7% | 112 |
 
@@ -86,7 +86,7 @@ paired for gaps. Measured by us on 28 October 2026.
 - **Against IndicXlit's full package: -2.8 points** (-3.6 to -2.1). The package is more accurate if you can afford its
   word list.
 - **Hinglish chat set:** 122 words people type in chat (everyday words, English words used in Hindi, brands, names),
-  written by us and checked word by word by a native Hindi speaker. Lekhani gets 112, level with IndicXlit's package.
+  written by us and checked word by word by a native Hindi speaker. Lekhani gets 113, one more than IndicXlit's package (112).
   The set is too small to call a few words' difference.
 - **Other open transliterators** we ran on the same test, word by word, scored far lower: IB-Emper
   hindi-translit-v3 (15M parameters, its model on its own) 48.6%, UnimeType Transliteration 4B 24.6% (v1.0) and
@@ -98,9 +98,8 @@ paired for gaps. Measured by us on 28 October 2026.
 ## Where it falls short
 
 - **About one word in three on this test is wrong at top-1.** The test is hard: many rare words and names. On
-  everyday chat words it does much better (41 of 44 in our chat set), but it still makes mistakes there.
-- **Some common words come out wrong:** `acha` → अच् (should be अच्छा; `accha` works), `kal` → काल (कल is its second
-  choice), `delhi` → देल्ही (should be दिल्ली).
+  everyday chat words it does much better (42 of 44 in our chat set), but it still makes mistakes there.
+- **Some common words come out wrong:** `kal` → काल (कल is its second choice), `delhi` → देल्ही (should be दिल्ली).
 - **No sentence context.** It writes one word at a time, so `main` ("I") and `mein` ("in") both become में:
   `main ghar ja raha hoon` → में घर जा रहा हूं (should be मैं).
 - **IndicXlit's full package is 2.8 points more accurate** on the main test.
@@ -125,6 +124,11 @@ human check.
 
 ## Changes
 
+- **1.2.0 (4 Nov 2026):** common words cut short are fixed. When the best spelling ends in a bare virama (्), which a
+  Hindi word almost never does, the runner now takes the best candidate without one, also trying a few spelling
+  variants of the input (for example `acha` → `accha`) and keeping only listed words. `acha` and `achha` now give
+  अच्छा. It changes about 0.1% of words: the Aksharantar Hindi test is unchanged at 65.9% (one word fixed, one broken),
+  and the Hinglish chat set rises from 112 to 113. Same model and word list; speed is unchanged.
 - **1.1.0 (2 Nov 2026):** the runner is about 10 times faster (batched beam search, WebAssembly SIMD kernels with a
   plain-JavaScript fallback, a GELU table, leaner beam search and a 2,000-word cache), contributed by Manit Roy
   ([GitHub #1](https://github.com/udgamlabs/lekhani/pull/1)). Same model and word list: the same best word and the same 8

translit.js

@@ -521,7 +521,7 @@ export class Transliterator {
     return res;
   }
 
-  rank(src) {
+  rank(src, noFix = false) {
     const { lam = 0, off = 0, beam = 4 } = this.rr, nb = this.nbest(src, beam);
     for (const c of nb) {
       const lp = this.list ? this.list.logp(c.text) : undefined;
@@ -530,7 +530,18 @@ export class Transliterator {
     }
     let best = null;
     for (const c of nb) if (!best || c.final > best.final) best = c;
-    return { best: best ? best.text : "", nbest: nb, candidates: [...nb].sort((a, b) => b.final - a.final) };
+    // A Hindi word almost never ends in a bare virama, so a best spelling that does was cut short (acha -> अच्).
+    // Then take the best candidate without one, from this beam plus the in-list candidates of a few spelling variants
+    // (acha -> accha, a final a added or lengthened). Fires on about 0.1% of words (Aksharantar Hindi, 4 Nov 2026).
+    let candidates = [...nb].sort((a, b) => b.final - a.final);
+    if (best && best.text.endsWith(VIRAMA) && !noFix) {
+      const pool = nb.filter((c) => !c.text.endsWith(VIRAMA));
+      for (const v of spellingVariants(src)) for (const c of this.rank(v, true).nbest) if (c.inList && !c.text.endsWith(VIRAMA)) pool.push(c);
+      for (const c of pool) if (best.text.endsWith(VIRAMA) || c.final > best.final) best = c;
+      // the chosen spelling leads the candidates, as it always does
+      candidates = [best, ...candidates.filter((c) => c.text !== best.text)];
+    }
+    return { best: best ? best.text : "", nbest: nb, candidates };
   }
 
   // a line of text: runs of Latin letters are transliterated, everything else is kept
@@ -539,6 +550,19 @@ export class Transliterator {
   }
 }
 
+const VIRAMA = "\u094d";
+
+// romanised spellings to try when the model cuts a word short: doubled middle consonant (acha -> accha), a final "a"
+// after a consonant, a final "a" lengthened
+function spellingVariants(w) {
+  const out = new Set(), m = w.match(/^(.*[aeiou])([bcdfgjklmnpqrstvwxyz])(h?)([aeiou]+)$/);
+  if (m) out.add(m[1] + m[2] + m[2] + m[3] + m[4]);
+  if (/[bcdfghjklmnpqrstvwxyz]$/.test(w)) out.add(w + "a");
+  if (/[^a]a$/.test(w)) out.add(w + "a");
+  out.delete(w);
+  return [...out];
+}
+
 function norm(s) {
   return s.normalize("NFC").replace(/[‌‍]/g, "");
 }