executorch-community/Whisper-Base-ExecuTorch-WebGPU

Whisper base for ExecuTorch WebGPU

This repository contains an ExecuTorch conversion of openai/whisper-base for the ExecuTorch WebGPU backend, so the model runs entirely in a web browser on the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on executorch.ai).

• Base model: openai/whisper-base
• Format: ExecuTorch .pte programs delegated to the WebGPU backend
• Quantization: mixed: int8 output layer (int8 per-channel weights, int8 activations with a calibrated static scale) and int8 token-embedding table; mel front end, encoder and decoder in fp32
• Total size: 229.6 MB

Note: OpenAI released Whisper's code under MIT; the weights are distributed under Apache-2.0, as on the upstream Hugging Face model card.

About the original model

The multilingual base Whisper model for speech recognition and translation, trained on 680k hours of labelled speech.

• Developed by: OpenAI
• Paper: Robust Speech Recognition via Large-Scale Weak Supervision
• 74M parameters
• Multilingual

For training data, evaluation results and full details, see the original model card: openai/whisper-base.

Intended use and limitations

Transcribing speech to text.

Accuracy varies by language, accent and recording conditions, and the model can hallucinate text that was not spoken, especially on silence or noise. Do not use it to transcribe people without their consent; see the upstream card's Performance and Limitations section.

What changed in this conversion

• Exported as six programs (mel front end, encoder, cross-attention K/V, two decoder forms, LM head); this repo ships the five used for streaming decoding. The full-recompute decoder (whisper_decoder_split.pte) is built by the recipe but not included
• Mel front end, encoder, cross-attention K/V projections and decoder run in fp32
• LM head: int8 weights (per output channel, symmetric) and int8 activations with a static per-tensor scale calibrated on 3 LibriSpeech clips that are not part of the evaluation set
• Token embedding table stored as int8 per row (row absmax / 127) with fp32 scales; the positional table is fp32
• Log-mel normalization runs on the host, between whisper_mel.pte (power mel spectrogram) and whisper_encoder.pte
• The LM head is padded from 51,865 to 51,872 outputs; the 7 padding logits are -1e9
• On 70 short clips from the LibriSpeech clean validation set, greedy transcripts have the same word error rate as transformers fp32 greedy decoding (8.42%), and 54 of 70 are token-identical to it. Only English transcription was tested

Inputs and outputs

• 16 kHz mono audio in windows of up to 30 s: zero-pad to 480,000 samples, reflect-pad 200 samples on each side, and cut 3000 frames of 400 samples at hop 160
• whisper_mel.pte: frames [1,3000,400] -> power mel spectrogram [1,3000,80] (Hann window, DFT and mel filters inside the program). On the host: x = log10(max(p, 1e-10)), x = max(x, max(x) - 8), x = (x + 4) / 4, then transpose to [1,80,3000]
• whisper_encoder.pte: log-mel [1,80,3000] -> hidden states [1,1500,512]. whisper_cross_kv.pte: hidden states -> 12 tensors [1,8,1500,64] (k0, v0, ..., k5, v5), computed once per window
• whisper_decode_stream.pte: forward(embedding [1,1,512], input_pos int64 [1], 12 cross K/V tensors) -> hidden [1,1,512], one token per call; the self-attention cache (448 positions) lives inside the program. whisper_lmhead.pte: hidden -> logits [1,1,51872]; take the argmax over the first 51,865
• Embedding input: row id of whisper_embed.i8.bin (int8 [51865,512]) times whisper_embed.scale.f32.bin[id] (float32 [51865]), plus row p of whisper_pos_embed.f32.bin (float32 [448,512]). English prefix: 50258, 50259, 50359, 50363; stop at 50257. Token text from vocab.json

Files

File	Size	SHA-256
vocab.json	816.0 KB	8f680bba319e01a6…
whisper_cross_kv.pte	12.0 MB	6e8078924168c21a…
whisper_decode_stream.pte	85.1 MB	bc2f33ad7aa16405…
whisper_embed.i8.bin	25.3 MB	bc7f8d8336ad935e…
whisper_embed.scale.f32.bin	202.6 KB	1f58851911611b96…
whisper_encoder.pte	78.6 MB	841e7702ba8fa735…
whisper_lmhead.pte	25.9 MB	68577fa240d29923…
whisper_mel.pte	694.3 KB	7cae318294879584…
whisper_pos_embed.f32.bin	896.0 KB	88fa1cdbf2b06f86…

Full checksums are in manifest.json. The playground verifies them after download.

Running it

These programs were exported with the ExecuTorch WebGPU backend (backends/webgpu) and run in a browser (headless Chromium, WebGPU on SwiftShader) with a development build of the ExecuTorch WebGPU WebAssembly runtime, where every program matched the CPU results (max abs error 1.6e-3 or less) and 3 test transcripts were token-identical to the CPU pipeline. That runtime build is not published yet; until it is, the playground is the supported way to run them. It downloads these files once, caches them in the browser, and runs everything locally.

Requirements: a browser with WebGPU enabled.

Provenance

• Upstream: openai/whisper-base at revision e37978b90ca9030d5170a5c07aadb050351a65bb (official)
• Exported with: executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git 3b6068392324)
• Reproduce: python recipes/whisper-base.py --out out/whisper-base/fp32dec-lmq8 --precision fp32dec-lmq8 (recipe in export/)
• Re-running the recipe produces byte-identical files.

Citation

If you use this model, please cite the original work:

@misc{radford2022whisper,
  doi={10.48550/ARXIV.2212.04356},
  url={https://arxiv.org/abs/2212.04356},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher={arXiv},
  year={2022}
}

License

This is a derivative of the upstream model and is distributed under the same terms (apache-2.0); see LICENSE. Modifications (conversion to the ExecuTorch format and quantization) by Meta Platforms, Inc.