lipi-hn

Small, fast, on-device speech recognition for English and Hinglish (Hindi and English mixed). Output is Roman script. Streaming-capable, 4-bit weights, 14.4 MB, built for CPU-only runtimes. For English only, see hitl-systems/lipi.

Results

Word error rate (%), lower is better. Greedy decoding, no language model, full context. Measured from the exported 4-bit weights with 8-bit activations.

English: the hf-audio/open-asr-leaderboard test configurations, cleaned versions where the leaderboard provides them.

Benchmark (test set)	WER, greedy	WER, with lm.bin
LibriSpeech test-clean	4.93	4.25
LibriSpeech test-other	12.02	10.65
Earnings-22 (cleaned, call level)	17.51	16.92 *
GigaSpeech (cleaned)	15.63	14.77
SPGISpeech	4.34	4.07
AMI (cleaned)	14.36	13.75
VoxPopuli (cleaned)	6.90	6.73
Mean of the seven	10.81	10.16

Language-model numbers come from beam search with lm.bin (alpha 0.1, beta 0.0) over the full test sets, measured on the 4-bit model's outputs before packing (greedy for the same outputs: 10.76 mean). * The Earnings-22 value applies the gain measured on its chunk-level outputs to the call-level greedy score; it is not a call-level rescore.

Hinglish and Hindi: the frozen golden splits of addyo07/noisy-hinglish-asr (hn_gold code-switched, hi_gold Hindi, hi_read read Hindi, and the noisy variants hn_goldn, hi_goldn). References are Devanagari or mixed script; we convert them to Roman with our own converter, so absolute numbers include its spelling choices. pWER ignores spelling variants of the same sound and is the headline metric.

Split	WER	pWER	CER
hn_gold (Hinglish, 487 utterances)	30.0	29.0	16.8
hn_goldn (noisy Hinglish)	43.2	42.3	27.2
hi_gold (Hindi, 361)	24.7	23.5	10.8
hi_goldn (noisy Hindi)	39.5	38.6	21.8
hi_read (read Hindi, 200)	21.2	19.7	8.5

With lm.bin and the same setting (alpha 0.1, beta 0.0), WER on the held-out half of hn_gold, hi_gold and hi_read falls by 8.6 % relative on average (the LM was never tuned on that half).

Comparison with small public Hinglish models

Same audio, same scorer, same Roman converter for every row; pWER (%), lower is better. The two comparison models are public Whisper fine-tunes that output Roman script: whisper-base-hinglish (73M parameters, MIT) and whisper-tiny-hinglish (38M parameters, MIT). They were decoded greedily with the settings in their model cards (language forced to en) on a GPU.

Split	lipi-hn	whisper-base-hinglish	whisper-tiny-hinglish
hn_gold	29.0	39.0	68.8
hn_goldn	42.3	45.8	64.3
hi_gold	23.5	49.0	68.0
hi_goldn	38.6	69.0	72.8
hi_read	19.7	39.2	56.1
Weights on disk	14.4 MB	290 MB (76 MB int8 build)	151 MB
CPU real-time factor, 4 threads	about 0.016 (PyTorch, different harness)	0.30	0.29

Words that are English inside Hinglish are the exception: on hn_gold the Whisper models make fewer errors on those words (about 22 % against 26 % for lipi-hn), while lipi-hn is better on Hindi words (27 % against 33 % and 39 %). whisper-tiny's score is dominated by inserted text (about 39 extra words per 100 on hn_gold). Single run; the other models were not tuned for these splits; English accuracy of the other models was not measured; the speed row compares two different measurement setups and is indicative only.

Using the language model

lm.bin is optional and needs a CTC beam decoder (for example pyctcdecode with the SentencePiece pieces as labels): alpha 0.1, beta 0.0, beam 16. Beam search is slower than greedy decoding.

Scoring: our own scorer; English references use the Whisper English text normaliser. This is not the leaderboard's own harness, so an official run may differ slightly. Known weak spots: long spontaneous code-switched speech, noisy or phone-quality audio, accented speech, and rare English words written in Devanagari (their Roman spelling is a convention).

Files

• lipi-hn-w4.lipi the 4-bit model (14.4 MB)
• tokenizer.model SentencePiece tokenizer (2048 pieces)
• lm.bin optional word language model (KenLM trie, 7.7 MB)

License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC BY-NC-SA 4.0) for the model weights and these documents: attribution required, non-commercial use only, and adaptations must be shared under the same licence. Full text in LICENSE.