Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,77 @@
+---
+license: apache-2.0
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- automatic-speech-recognition
+base_model:
+- openai/whisper-small
+---
+# Whisper-small — ExecuTorch (encoder + decoder)
+
+Speech recognition in two `.pte` files: the encoder runs once per 30-second window, the
+decoder once per generated token. Putting them in one graph would re-encode the audio on
+every step.
+
+| graph | build | file | size (MB) | corr vs fp32 eager | ms | eager ms |
+|---|---|---|---|---|---|---|
+| encoder | XNNPACK fp32 | `whisper_small_encoder_xnnpack_fp32.pte` | 352.8 | 1.000000 | 357.4 | 144.7 |
+| encoder | XNNPACK fp16 | `whisper_small_encoder_xnnpack_fp16.pte` | 180.5 | 1.000000 | 619.5 | 144.4 |
+| encoder | XNNPACK int8 | `whisper_small_encoder_xnnpack_int8.pte` | 98.3 | 0.998973 | 333.2 | 144.1 |
+| encoder | Core ML | `whisper_small_encoder_coreml_all.pte` | 176.8 | 0.999942 | 95.5 | 143.4 |
+| decoder | XNNPACK fp32 | `whisper_small_decoder_xnnpack_fp32.pte` | 774.0 | 1.000000 | 91.2 | 55.3 |
+| decoder | XNNPACK fp16 | `whisper_small_decoder_xnnpack_fp16.pte` | 387.3 | 0.999994 | 188.6 | 55.3 |
+| decoder | Core ML | `whisper_small_decoder_coreml_all.pte` | 307.6 | 0.999863 | 13.7 | 55.6 |
+
+Every file takes and returns fp32 tensors (token ids stay int64), so any encoder pairs with
+any decoder. The lightest working pair is 405.9 MB.
+
+- **Source**: [openai/whisper-small](https://huggingface.co/openai/whisper-small)
+- **License**: Apache-2.0
+- **Encoder input**: log-mel spectrogram `[1, 80, 3000]` — 30 s at 16 kHz, 80 mel bins, hop
+  160, window 400, exactly what `WhisperFeatureExtractor` produces
+- **Decoder input**: the encoder output plus `decoder_input_ids [1, 128]` int64,
+  left-aligned and padded. Start with `<|startoftranscript|>`, a language token,
+  `<|transcribe|>`, `<|notimestamps|>`.
+
+## Decoding
+
+No KV cache: the decoder is a static graph over a fixed 128-token window, so a greedy step
+is take `argmax` of row `len-1`, append it, run again. Stop at `<|endoftext|>` (50257). 128
+tokens covers a 30-second window of ordinary speech; past that, start a new window.
+
+That costs a full 128-position forward pass per token, which is the price of a static graph
+that runs unchanged across runtimes and precisions.
+
+## Verification (Mac arm64, executorch 1.4.0, torch 2.13.0)
+
+The two wrappers compose back to `WhisperForConditionalGeneration` exactly — max_abs_diff
+**0.000e+00** — and every graph matches torch fp32 eager at the correlations above. Timings
+are medians over 5 runs in one process: a relative reference, not a device number.
+
+## Two things worth knowing about the sizes
+
+**The decoder `.pte` is larger than the decoder's weights.** Whisper ties `proj_out.weight`
+to `decoder.embed_tokens.weight`, but the two uses need different representations: an
+embedding table the portable kernels index into, and the same values packed into the XNNPACK
+delegate's blob for the output matmul. Tying them in PyTorch does not tie them here.
+Referencing the weight through `F.linear` instead of the `proj_out` module does not either —
+exported both ways, whisper-tiny's decoder comes out at 198.0 MB exactly.
+
+**The decoder has no int8 build.** PT2E puts an observer on the int64 `decoder_input_ids`
+feeding the token embedding, and the lookup then refuses a float index (`tensors used as
+indices must be long, int, byte or bool`). The encoder takes float mel and quantizes without
+complaint, which is where the size is worth taking anyway.
+
+## Conversion
+
+```bash
+python convert/export_whisper.py small
+```
+
+The ExecuTorch tree ships a single-graph Whisper example under `examples/models/whisper`;
+this is that model with the halves separated.
+
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))