| --- |
| license: apache-2.0 |
| language: |
| - en |
| - de |
| - fr |
| - es |
| - it |
| - pt |
| - nl |
| - pl |
| - ta |
| - hi |
| - te |
| - kn |
| - ml |
| tags: |
| - automatic-speech-recognition |
| - whisper |
| - sherpa-onnx |
| - onnx |
| - on-device |
| - quantized |
| library_name: sherpa-onnx |
| pipeline_tag: automatic-speech-recognition |
| base_model: |
| - openai/whisper-tiny |
| - vasista22/whisper-tamil-small |
| - vasista22/whisper-hindi-small |
| - vasista22/whisper-telugu-tiny |
| - vasista22/whisper-kannada-tiny |
| - kavyamanohar/whisper-small-malayalam |
| --- |
| |
| # indic-asr-onnx |
|
|
| INT8-quantized Whisper models in [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) |
| format, packaged for on-device speech recognition. |
|
|
| ## Packs |
|
|
| Each pack contains `encoder.int8.onnx`, `decoder.int8.onnx`, and `tokens.txt`, |
| and is downloaded independently at runtime. |
|
|
| | Pack | Languages | Size | Source model | |
| |---|---|---|---| |
| | `whisper-tiny` | en, de, fr, es, it, pt, nl, pl (+90 more) | 99 MB | [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny) | |
| | `whisper-small-ta` | Tamil | 358 MB | [vasista22/whisper-tamil-small](https://huggingface.co/vasista22/whisper-tamil-small) | |
| | `whisper-small-hi` | Hindi | 358 MB | [vasista22/whisper-hindi-small](https://huggingface.co/vasista22/whisper-hindi-small) | |
| | `whisper-small-ml` | Malayalam | 358 MB | [kavyamanohar/whisper-small-malayalam](https://huggingface.co/kavyamanohar/whisper-small-malayalam) | |
| | `whisper-tiny-te` | Telugu | 99 MB | [vasista22/whisper-telugu-tiny](https://huggingface.co/vasista22/whisper-telugu-tiny) | |
| | `whisper-tiny-kn` | Kannada | 99 MB | [vasista22/whisper-kannada-tiny](https://huggingface.co/vasista22/whisper-kannada-tiny) | |
|
|
| `whisper-tiny` is the multilingual base model — it serves all European languages |
| from a single download. The Indic packs are language-specific fine-tunes, which |
| substantially outperform stock Whisper on those languages. |
|
|
| ## Structure |
|
|
| ``` |
| models/ |
| ├── whisper-tiny/ |
| │ ├── encoder.int8.onnx |
| │ ├── decoder.int8.onnx |
| │ └── tokens.txt |
| ├── whisper-small-ta/ |
| ├── whisper-small-hi/ |
| ├── whisper-small-ml/ |
| ├── whisper-tiny-te/ |
| └── whisper-tiny-kn/ |
| ``` |
|
|
| ## Usage |
|
|
| ### Python (sherpa-onnx) |
|
|
| ```python |
| import sherpa_onnx |
| |
| recognizer = sherpa_onnx.OfflineRecognizer.from_whisper( |
| encoder="models/whisper-small-ta/encoder.int8.onnx", |
| decoder="models/whisper-small-ta/decoder.int8.onnx", |
| tokens="models/whisper-small-ta/tokens.txt", |
| language="ta", |
| task="transcribe", |
| num_threads=2, |
| ) |
| |
| stream = recognizer.create_stream() |
| stream.accept_waveform(16000, samples) # float32 PCM, 16 kHz mono |
| recognizer.decode_stream(stream) |
| print(stream.result.text) |
| ``` |
|
|
| ### Android (Kotlin) |
|
|
| ```kotlin |
| val config = OfflineRecognizerConfig( |
| featConfig = FeatureConfig(sampleRate = 16000, featureDim = 80), |
| modelConfig = OfflineModelConfig( |
| whisper = OfflineWhisperModelConfig( |
| encoder = "$packDir/encoder.int8.onnx", |
| decoder = "$packDir/decoder.int8.onnx", |
| language = "ta", |
| task = "transcribe", |
| ), |
| tokens = "$packDir/tokens.txt", |
| numThreads = 2, |
| ) |
| ) |
| val recognizer = OfflineRecognizer(config) |
| ``` |
|
|
| ## Export |
|
|
| Exported with sherpa-onnx's |
| [`scripts/whisper/export-onnx.py`](https://github.com/k2-fsa/sherpa-onnx/blob/master/scripts/whisper/export-onnx.py), |
| then dynamically quantized to INT8. |
|
|
| HuggingFace fine-tunes were first converted to OpenAI Whisper checkpoint format |
| (`{"dims": ..., "model_state_dict": ...}`) so the export script could load them. |
|
|
| Note: on PyTorch ≥ 2.9 the export script requires `dynamo=False` to be passed to |
| `torch.onnx.export` — the dynamo-based exporter fails on Whisper's |
| data-dependent positional embedding indexing. |
|
|
| ## Credits |
|
|
| - Fine-tuned Indic models by **[vasista22](https://huggingface.co/vasista22)** |
| (Speech Lab, IIT Madras — funded by Bhashini, MeitY, Govt. of India) and |
| **[kavyamanohar](https://huggingface.co/kavyamanohar)** |
| - Base models by **OpenAI** ([Whisper](https://github.com/openai/whisper)) |
| - Export tooling by **[k2-fsa](https://github.com/k2-fsa)** (sherpa-onnx) |
|
|
| ## License |
|
|
| Apache 2.0, inherited from the upstream models. |
|
|
