Upload README.md with huggingface_hub

README.md

@@ -19,10 +19,11 @@ every step.
 |---|---|---|---|---|---|---|
 | encoder | XNNPACK fp32 | `whisper_small_encoder_xnnpack_fp32.pte` | 352.8 | 1.000000 | 357.4 | 144.7 |
 | encoder | XNNPACK fp16 | `whisper_small_encoder_xnnpack_fp16.pte` | 180.5 | 1.000000 | 619.5 | 144.4 |
-| encoder | XNNPACK int8 | `whisper_small_encoder_xnnpack_int8.pte` | 98.3 | 0.998973 | 333.2 | 144.1 |
+| encoder | XNNPACK int8 | `whisper_small_encoder_xnnpack_int8.pte` | 98.3 | 0.999052 | 339.3 | 151.8 |
 | encoder | Core ML | `whisper_small_encoder_coreml_all.pte` | 176.8 | 0.999942 | 95.5 | 143.4 |
 | decoder | XNNPACK fp32 | `whisper_small_decoder_xnnpack_fp32.pte` | 774.0 | 1.000000 | 91.2 | 55.3 |
 | decoder | XNNPACK fp16 | `whisper_small_decoder_xnnpack_fp16.pte` | 387.3 | 0.999994 | 188.6 | 55.3 |
+| decoder | XNNPACK int8 | `whisper_small_decoder_xnnpack_int8.pte` | 315.5 | 0.994395 | 85.2 | 57.6 |
 | decoder | Core ML | `whisper_small_decoder_coreml_all.pte` | 307.6 | 0.999863 | 13.7 | 55.6 |
 
 Every file takes and returns fp32 tensors (token ids stay int64), so any encoder pairs with
@@ -60,10 +61,17 @@ delegate's blob for the output matmul. Tying them in PyTorch does not tie them h
 Referencing the weight through `F.linear` instead of the `proj_out` module does not either —
 exported both ways, whisper-tiny's decoder comes out at 198.0 MB exactly.
 
-**The decoder has no int8 build.** PT2E puts an observer on the int64 `decoder_input_ids`
-feeding the token embedding, and the lookup then refuses a float index (`tensors used as
-indices must be long, int, byte or bool`). The encoder takes float mel and quantizes without
-complaint, which is where the size is worth taking anyway.
+**The decoder's int8 build is the smallest portable one.** Dynamic int8 quantizes the linear weights and
+leaves the token embedding table in fp32, and that table is 159.3 MB — 51,865 tokens at
+768 dimensions. On this size the rest of the decoder finally outweighs it: int8 comes in at 315.5 MB against fp16's 387.3 MB, so it is the smallest portable build and it ships.
+
+Until recently there was no decoder int8 build at all, and this card said PT2E was observing
+the int64 `decoder_input_ids`. That was wrong. `XNNPACKQuantizer.transform_for_annotation`
+rewrites every scalar argument of `add.Tensor`/`mul.Tensor` as `torch.tensor(float(arg))`
+whatever the node's dtype, so the `arange(n) + 0` that builds the attention mask came back
+float32 and could no longer index — one line in ExecuTorch's
+`backends/xnnpack/quantizer/xnnpack_quantizer_utils.py`, still present on main, and nothing
+to do with the token ids.
 
 ## Conversion