| --- |
| library_name: llima |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| base_model: openai/whisper-medium |
| base_model_relation: quantized |
| language: |
| - en |
| - zh |
| - de |
| - es |
| - ru |
| - ko |
| - fr |
| - ja |
| - pt |
| - tr |
| - pl |
| - ca |
| - nl |
| - ar |
| - sv |
| - it |
| - id |
| - hi |
| - fi |
| - vi |
| - he |
| - uk |
| - el |
| - ms |
| - cs |
| - ro |
| - da |
| - hu |
| - ta |
| - "no" |
| - th |
| - ur |
| - hr |
| - bg |
| - lt |
| - la |
| - mi |
| - ml |
| - cy |
| - sk |
| - te |
| - fa |
| - lv |
| - bn |
| - sr |
| - az |
| - sl |
| - kn |
| - et |
| - mk |
| - br |
| - eu |
| - is |
| - hy |
| - ne |
| - mn |
| - bs |
| - kk |
| - sq |
| - sw |
| - gl |
| - mr |
| - pa |
| - si |
| - km |
| - sn |
| - yo |
| - so |
| - af |
| - oc |
| - ka |
| - be |
| - tg |
| - sd |
| - gu |
| - am |
| - yi |
| - lo |
| - uz |
| - fo |
| - ht |
| - ps |
| - tk |
| - nn |
| - mt |
| - sa |
| - lb |
| - my |
| - bo |
| - tl |
| - mg |
| - as |
| - tt |
| - haw |
| - ln |
| - ha |
| - ba |
| - jw |
| - su |
| tags: |
| - whisper |
| - audio |
| - automatic-speech-recognition |
| - speech-translation |
| - embedded |
| - sima |
| - modalix |
| - a16w8 |
| --- |
| |
| # Whisper Medium A16W8: Optimized for SiMa.ai Modalix |
|
|
| ## Overview |
|
|
| This repository contains a precompiled and quantized version of |
| [OpenAI Whisper Medium](https://huggingface.co/openai/whisper-medium), optimized |
| for execution on the **SiMa.ai Modalix** platform. |
|
|
| | Property | Value | |
| |---|---| |
| | Model architecture | Whisper encoder-decoder Transformer | |
| | Parameters | 769 million | |
| | Quantization | A16W8: 16-bit activations, 8-bit weights | |
| | Target | SiMa.ai Modalix | |
| | Tasks | Multilingual transcription and speech translation to English | |
| | Languages | 99 | |
| | Maximum audio window | 30 seconds | |
| | Maximum decoder positions | 448 | |
| | Compatible Neat release | 0.4.0 | |
| | Source model | [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) | |
|
|
| > This repository is not a standard Transformers checkpoint. The `elf_files/` |
| > and `devkit/` directories contain compiled artifacts consumed by the SiMa.ai |
| > Neat runtime on Modalix. |
| |
| ## Prerequisites |
| |
| To use this model, you need: |
| |
| 1. A **SiMa.ai Modalix device**. |
| 2. The **SiMa.ai Neat Runtime** installed or updated on Modalix. |
| 3. The **Hugging Face CLI**, optionally, for downloading the model on a host |
| before copying it to Modalix. |
| |
| Follow the [SiMa.ai Neat getting started guide](https://developer.sima.ai/software/getting-started/) |
| to install or update the Neat Runtime. |
| |
| ## Installation and Deployment |
| |
| ### Download the Model |
| |
| On Modalix, download the precompiled model with the LLiMa model manager: |
| |
| ```bash |
| llima pull whisper-medium-a16w8 |
| ``` |
| |
| The model is installed under: |
| |
| ```text |
| /media/nvme/llima/models/whisper-medium-a16w8 |
| ``` |
| |
| Alternatively, download it on a host and copy it to Modalix: |
| |
| ```bash |
| hf download simaai/whisper-medium-a16w8 \ |
| --local-dir whisper-medium-a16w8 |
| |
| scp -r whisper-medium-a16w8 \ |
| sima@<modalix-ip>:/media/nvme/llima/models/ |
| ``` |
| |
| Replace `<modalix-ip>` with the IP address or hostname of the Modalix device. |
| |
| ## Usage |
| |
| This model is intended for applications built with the SiMa.ai Neat GenAI |
| APIs. Use `ASRModel` for direct, in-process transcription or `GenAIServer` when |
| clients access the model over HTTP. |
| |
| ### Direct Application Integration |
| |
| The Neat `ASRModel` API provides the lowest-overhead path for speech |
| transcription inside an application: |
| |
| ```cpp |
| #include "neat/genai.h" |
| |
| #include <iostream> |
| |
| int main() { |
| simaai::neat::genai::ASRModel model( |
| "/media/nvme/llima/models/whisper-medium-a16w8"); |
| |
| simaai::neat::genai::GenerationRequest request; |
| request.audio_file = "speech.wav"; |
| request.language = "en"; |
| |
| auto result = model.run(request); |
| std::cout << result.text << "\n"; |
| } |
| ``` |
| |
| For the complete C++ and Python workflows, including audio tensors and |
| streaming, see |
| [GenAI Model](https://developer.sima.ai/software/develop-apps/development-workflow/genai-model). |
|
|
| ### Serve with the Neat GenAI Server |
|
|
| Use `GenAIServer` when a browser, service, or remote client needs to access the |
| model through an HTTP API. The packaged Neat tutorial can serve this ASR model |
| by itself: |
|
|
| ```bash |
| ./lib/sima-neat/tutorials/tutorial_021_serve_genai_models \ |
| --asr /media/nvme/llima/models/whisper-medium-a16w8 |
| ``` |
|
|
| Verify that the model was registered: |
|
|
| ```bash |
| curl http://<modalix-ip>:9998/v1/models |
| ``` |
|
|
| Send an audio transcription request with the tutorial client: |
|
|
| ```bash |
| python3 share/sima-neat/tutorials/021_serve_genai_models/request_audio_transcription.py \ |
| --server-ip <modalix-ip> \ |
| --model asr \ |
| speech.wav |
| ``` |
|
|
| See |
| [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models) |
| for the complete C++, Python, and HTTP workflow. |
|
|
| ## Artifact Layout |
|
|
| ```text |
| devkit/ Runtime configuration, tokenizer, and embeddings |
| elf_files/ Compiled Modalix accelerator programs |
| compile.sh Compilation entry point |
| gen_models--openai--whisper.py Whisper compilation utility |
| ``` |
|
|
| The compilation scripts are included for provenance and reproducibility. They |
| are not required when using the precompiled artifacts. |
|
|
| ## Evaluation |
|
|
| No accuracy or performance results specific to this A16W8 compiled artifact |
| are currently reported. Results on the |
| [source model card](https://huggingface.co/openai/whisper-medium) describe the |
| unquantized upstream checkpoint and should not be treated as measurements of |
| this build. |
|
|
| ## Limitations |
|
|
| - These artifacts run on SiMa.ai Modalix and cannot be loaded with |
| `transformers.AutoModel`. |
| - The runtime processes an audio window of up to 30 seconds. Segment longer |
| recordings before inference. |
| - This build produces text without word- or segment-level timestamps. |
| - Speaker diarization is not provided. |
| - Quantization can cause minor differences from the full-precision source |
| model. |
| - Whisper can hallucinate text during silence or noisy audio and performs |
| unevenly across languages, accents, and domains. Evaluate the model on |
| representative data before production deployment. |
|
|
| ## Resources |
|
|
| - [SiMa.ai Neat](https://developer.sima.ai/software/getting-started/) |
| - [GenAI Model API](https://developer.sima.ai/software/develop-apps/development-workflow/genai-model) |
| - [Serve GenAI Models](https://developer.sima.ai/software/tutorials/serve-genai-models) |
| - [GenAI with LLiMa](https://developer.sima.ai/software/genai-llima/) |
| - [OpenAI Whisper Medium](https://huggingface.co/openai/whisper-medium) |
| - [Whisper paper](https://arxiv.org/abs/2212.04356) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{radford2022whisper, |
| doi = {10.48550/ARXIV.2212.04356}, |
| url = {https://arxiv.org/abs/2212.04356}, |
| author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and |
| Brockman, Greg and McLeavey, Christine and Sutskever, Ilya}, |
| title = {Robust Speech Recognition via Large-Scale Weak Supervision}, |
| publisher = {arXiv}, |
| year = {2022} |
| } |
| ``` |
|
|
