README.md

metadata

license: apache-2.0
language:
  - multilingual
pipeline_tag: automatic-speech-recognition
tags:
  - audio
  - automatic-speech-recognition
  - whisper
  - safetensors

Whisper inference weights

Inference-oriented conversions of OpenAI Whisper checkpoints. The model architecture and learned parameters come from OpenAI Whisper and the corresponding openai/whisper-* repositories.

This repository uses branches to keep each model at the conventional root path model.safetensors:

Revision	Source	Parameters	Languages
tiny	openai/whisper-tiny	39M	Multilingual
small	openai/whisper-small	244M	Multilingual
medium	openai/whisper-medium	769M	Multilingual
large-v3	openai/whisper-large-v3	1.55B	Multilingual

Always select a revision. The main branch is an index and does not contain weights.

Each model branch also provides an experimental model-fp8.safetensors. Linear matrices use OCP E4M3 with one FP16 scale per output channel. Token and positional embeddings, convolutions, biases, and normalization parameters remain FP16/FP32. FP8 reduces storage but requires runtime support for the companion *.weight_scale tensors.

Transformation

The conversion is deterministic and does not train or otherwise alter the model:

• Linear, convolution, embedding, and other compute weights are stored as IEEE FP16.
• Positional embeddings and LayerNorm affine parameters remain FP32, matching Whisper's mixed-precision inference behavior.
• Hugging Face Transformers parameter paths are normalized to the concise OpenAI Whisper layout, for example model.decoder.layers.0.self_attn.q_proj.weight becomes decoder.blocks.0.attn.query.weight.
• Tied token embeddings are stored once in FP16 and used for both token lookup and vocabulary projection.
• Non-floating tensors are preserved without conversion.
• Tokenizer, generation, preprocessing, and architecture configuration files are copied from each source revision.

The exact converter is included as convert.py and can be run without PyTorch:

uv run --with numpy --with safetensors python convert.py \
  source/model.safetensors model.safetensors \
  --source openai/whisper-medium@abdf7c39ab9d0397620ccaea8974cc764cd0953e

Add --with ml-dtypes and --compute-dtype float8_e4m3fn to produce the scaled FP8 variant.

Revision	Converted size	Converted SHA-256
tiny	77 MB	bbdc114a81c7775baffe0d3f045e5ab591eb3a6e1ac00a480136ff770ea783ba
small	486 MB	b02cafebb1e736ea3968de06d96041a3dfc48a22820d74f5391b7daae5e36728
medium	1.53 GB	e6d6ede74e0f4fd5d00042ecc56b6c4efa9b9ad6cd93e870379e9cbd0c884435
large-v3	3.09 GB	c33cd318544b49c0586589c9b9bffa72a90561dac4cb59c76fc3c31314c98e95

Revision	FP8 size	FP8 SHA-256
tiny	60 MB	8d3845821e2ac5f8f6d9aa556481900d53fe3895436f217d664574295cead50c
small	289 MB	e9fd7e3418781a1b3468685522d90aeebe8df729ecb1e7ecbc2de4b9a9e8540c
medium	829 MB	fea31295a43f7ddbdc429dd5b7b59e5b7bebef9ee8aee7f69bfd243897c59638
large-v3	1.63 GB	9806d2a748c84841f516eba5c89b00c62664706ac635fda1685caa83fb040e6c

On AMD gfx1151, FP8 is emulated as FP16 because the architecture has no native FP8 WMMA path. It therefore saves storage but is slower: Medium took 8.90s versus 4.70s and changed a short phrase; Large V3 took 6.20s versus 2.74s while preserving the tested transcript. Use FP16 weights for speed on gfx1151.

Validation status: Large V3 preserved the tested transcript; Medium remained coherent but was not transcript-identical; Tiny produced repetitive output on the tested Russian clip; Small has only format/structure validation. All FP8 files are therefore marked experimental rather than parity-equivalent.

The mixed storage cuts checkpoint size and prevents runtimes from repeatedly converting a large FP32 vocabulary matrix during autoregressive decoding. FP16 conversion introduces the expected rounding relative to the source FP32 checkpoints.

Because parameter names are normalized, these weights are not a drop-in transformers checkpoint. Consumers must understand the OpenAI-style paths. Svod can load this layout directly.

Loading

With Svod, pass the branch as the Hub revision and use dimensions matching that branch:

let model = Whisper::from_hub(
    "vpermilp/whisper",
    "medium",
    ModelDimensions::for_size(WhisperSize::Medium),
)?;

Select the experimental FP8 file explicitly:

let model = Whisper::from_hub_with_weights(
    "vpermilp/whisper",
    "large-v3",
    "model-fp8.safetensors",
    ModelDimensions::for_size(WhisperSize::LargeV3),
)?;

With the inference example, use --weights model-fp8.safetensors.

For generic safetensors consumers:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "vpermilp/whisper",
    "model.safetensors",
    revision="medium",
)

Model

Whisper is a Transformer encoder-decoder model for multilingual speech recognition and speech translation to English. It was trained by OpenAI on 680,000 hours of weakly supervised audio data. See the paper Robust Speech Recognition via Large-Scale Weak Supervision, the original repository, and each source model card for architecture details, evaluation results, intended uses, training data, and complete limitations.

Limitations

These converted weights retain the source models' behavior and limitations. Whisper can hallucinate text not present in the audio, generate repetitions, perform unevenly across languages and accents, and make consequential transcription errors. Evaluate the selected model on the target language, domain, acoustic conditions, and hardware before deployment. Do not use transcription or inferred attributes for high-risk decisions, and do not transcribe people without an appropriate legal basis or consent.

No independent benchmark suite is claimed for these conversions. Results reported by OpenAI apply to the source checkpoints and evaluation procedures; FP16 rounding can produce small output differences.

License and attribution

The source checkpoints are distributed under the Apache License 2.0. This repository preserves that license and attributes OpenAI and the Whisper authors. Model-card content is adapted in part from the OpenAI/Hugging Face Whisper cards.

@article{radford2022robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2022}
}