File size: 5,261 Bytes
42dc386 8af3517 42dc386 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | ---
library_name: onnx
license: mit
tags:
- foundation
- amd
- rocm
- automatic-speech-recognition
pipeline_tag: automatic-speech-recognition
---
# Whisper: Optimized for AMD ROCm
OpenAI Whisper is a transformer encoder-decoder model for automatic speech recognition (ASR) — it transcribes audio into text via autoregressive decoding. This repository packages inference for speech recognition using **ONNX Runtime**, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs, CPUs, and NPUs.
This is based on the implementation of Whisper found [here](https://github.com/openai/whisper).
This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [whisper AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/whisper) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline).
---
## Task Overview
**Task:** Automatic speech recognition (transcription)
**Dataset:** Bundled 3-sample LibriSpeech smoke-test set for quick sanity checks; full-dataset accuracy uses [AudioLLMs/librispeech_test_clean](https://huggingface.co/datasets/AudioLLMs/librispeech_test_clean) (2,617 samples)
**Output metrics:** Word Error Rate (WER), Character Error Rate (CER)
> **Note:** The exported graph is two ONNX models — an encoder (mel spectrogram → hidden states) and an autoregressive decoder (hidden states → tokens).
---
## AMD ROCm Optimization
This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points:
- Validated backends: **ONNX Runtime** on CPU (FP32), GPU (MIGraphX — FP32/FP16), and NPU (VitisAI — FP32 ONNX input, internally quantized).
- No code changes required versus the upstream Whisper implementation — only environment/runtime configuration differs.
- GPU pays a one-time 30+ minute MIGraphX graph-compilation cost on first run; NPU pays a similar one-time VitisAI compilation cost.
| Runtime | Precision | Backend | Hardware | Notes |
|---|---|---|---|---|
| ONNX Runtime | FP32 | CPU Execution Provider | AMD CPU | — |
| ONNX Runtime | FP32 / FP16 | MIGraphX Execution Provider | AMD Instinct™ / Radeon™ GPU (ROCm) | — |
| ONNX Runtime | FP32 | VitisAI Execution Provider | AMD Ryzen AI NPU | Accepts FP32 ONNX input; VitisAI quantizes internally |
---
## Getting Started
For setup instructions, evaluation scripts, and custom configuration options, see the [whisper on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/whisper).
---
## Model Details
**Model Type:** Automatic speech recognition (transformer encoder-decoder)
**Base Model:** `openai/whisper-large-v3`
**Model Stats:**
- Exported as two ONNX graphs: encoder (mel spectrogram → hidden states) and autoregressive decoder (hidden states → tokens)
- Precision tested: FP32 (CPU, NPU); FP32, FP16 (GPU)
---
## Accuracy Pipeline
Lower WER/CER means better transcription accuracy — 0% is a perfect transcription (every word/character matches the reference), while 100% means the output shares nothing with it. In practice, well-tuned Whisper models score in the low single-digit percent range on LibriSpeech test-clean; scores above ~10% indicate significant transcription errors. Scoring uses Whisper-style text normalization (`transformers`' `BasicTextNormalizer`).
### Metrics Explained
| Metric | Description |
|--------|-------------|
| WER (Word Error Rate) | Word-level edit distance (insertions + deletions + substitutions) divided by the reference word count. Lower is better — a WER of 5% means roughly 1 in 20 words is wrong. |
| CER (Character Error Rate) | The same edit-distance computation applied at the character level instead of the word level. Lower is better, and typically smaller than WER for the same transcript since one misspelled character only counts as a single error. |
### Accuracy Results
**Results (LibriSpeech test-clean)**:
<!-- accuracy-table-start -->
| Run | Device | Precision | Samples | WER | CER |
|-----|--------|-----------|---------|-----|-----|
| Smoke test | CPU | FP32 | 3 | 0.114 | 0.025 |
| Full-dataset (subset) | CPU | FP32 | 25 / 2617 | 0.035 | 0.015 |
| Full-dataset (subset) | GPU | FP32 | 25 / 2617 | 0.035 | 0.015 |
<!-- accuracy-table-end -->
**Note:** These full-dataset rows are drawn from a 25-sample subset of the 2,617-sample LibriSpeech test-clean split — indicative only. A complete run over all 2,617 samples takes on the order of hours due to autoregressive greedy decoding.
---
## Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/whisper)**
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Encoder/decoder ONNX export and per-operator profiling scripts
- LibriSpeech test-clean dataset staging and WER/CER evaluation pipeline
- Benchmarking and reproduction instructions across CPU, GPU, and NPU
|