Add ExecuTorch WebGPU program for whisper-base

README.md

@@ -5,6 +5,7 @@ base_model:
 base_model_relation: quantized
 library_name: executorch
 pipeline_tag: automatic-speech-recognition
+inference: false
 tags:
 - executorch
 - webgpu
@@ -15,8 +16,7 @@ tags:
 
 This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
 [openai/whisper-base](https://huggingface.co/openai/whisper-base) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
-the visitor's GPU. It powers the corresponding demo in the
-ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
+the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
 
 - **Base model:** [openai/whisper-base](https://huggingface.co/openai/whisper-base)
 - **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
@@ -25,6 +25,34 @@ ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executor
 
 > **Note:** OpenAI released Whisper's code under MIT; the weights are distributed under Apache-2.0, as on the upstream Hugging Face model card.
 
+## About the original model
+
+The multilingual base Whisper model for speech recognition and translation, trained on 680k hours of labelled speech.
+
+- **Developed by:** OpenAI
+- **Paper:** [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
+- 74M parameters
+- Multilingual
+
+For training data, evaluation results and full details, see the original model card: [openai/whisper-base](https://huggingface.co/openai/whisper-base).
+
+## Intended use and limitations
+
+Transcribing speech to text.
+
+Accuracy varies by language, accent and recording conditions, and the model can hallucinate text that was not spoken, especially on silence or noise. Do not use it to transcribe people without their consent; see the upstream card's Performance and Limitations section.
+
+## What changed in this conversion
+
+- Token embeddings quantized to int8
+- Log-mel front end, encoder, cross-attention KV, decoder and LM head exported as separate programs for streaming decoding
+
+## Inputs and outputs
+
+- 16 kHz mono audio in Whisper's 30-second windows
+- `whisper_mel.pte`: audio to log-mel spectrogram; `whisper_encoder.pte`: spectrogram to audio features; `whisper_cross_kv.pte`: features to cross-attention keys and values
+- `whisper_decode_stream.pte` / `whisper_decoder_split.pte` with `whisper_lmhead.pte`: token-by-token decoding; token text from `vocab.json`
+
 ## Files
 
 | File | Size | SHA-256 |
@@ -44,16 +72,34 @@ Full checksums are in `manifest.json`. The playground verifies them after downlo
 
 ## Running it
 
-These programs need the ExecuTorch WebGPU runtime compiled to WebAssembly
-([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)). The easiest way to
-try the model is the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)),
-which downloads these files once, caches them in the browser, and runs everything locally.
+These programs were exported with the ExecuTorch WebGPU backend
+([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and validated with the
+WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). That runtime build is not published yet; until it is,
+the playground is the supported way to run them. It downloads these files once, caches them in the browser,
+and runs everything locally.
+
+Requirements: a browser with WebGPU enabled.
 
 ## Provenance
 
 - Converted from the upstream checkpoint by the ExecuTorch team with the WebGPU export path.
 - A reproducible export recipe will be published alongside the open-source ExecuTorch WebGPU runtime.
 
+## Citation
+
+If you use this model, please cite the original work:
+
+```bibtex
+@misc{radford2022whisper,
+  doi={10.48550/ARXIV.2212.04356},
+  url={https://arxiv.org/abs/2212.04356},
+  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
+  title={Robust Speech Recognition via Large-Scale Weak Supervision},
+  publisher={arXiv},
+  year={2022}
+}
+```
+
 ## License
 
 This is a derivative of the upstream model and is distributed under the same terms