Fix i8 variant: keep encoder float32 (quantized encoder was broken) (#2)

- Fix i8 variant: keep encoder float32 (quantized encoder was broken) (435b7e671b41230124c010ee318e10b70a4c9457)


Co-authored-by: Majima <mlboydaisuke@users.noreply.huggingface.co>

README.md

@@ -20,9 +20,9 @@ tags:
 It transcribes English speech and is designed for fast on-device inference.
 
 This repository packages the model for [LiteRT](https://ai.google.dev/edge/litert):
-a float32 model, a dynamic-range int8 model, and ahead-of-time compiled
-variants for a range of MediaTek and Qualcomm SoCs so the model can run on the
-device NPU.
+a float32 model, an int8 model (dynamic-range-quantized decoder with a
+float32 encoder), and ahead-of-time compiled variants for a range of MediaTek
+and Qualcomm SoCs so the model can run on the device NPU.
 
 ## Model description
 
@@ -49,7 +49,7 @@ transcription loop:
 | File | Description |
 |---|---|
 | `moonshine_tiny_5s_f32.tflite` | float32 model (109 MB) |
-| `moonshine_tiny_5s_i8.tflite` | dynamic-range int8 model (29 MB) |
+| `moonshine_tiny_5s_i8.tflite` | int8 model (52 MB): float32 encoder + dynamic-range int8 decoder |
 | `moonshine_tiny_5s_f32_MediaTek_*.tflite` | float32 AOT-compiled for MediaTek NPUs (per SoC) |
 | `moonshine_tiny_5s_f32_Qualcomm_*.tflite` | float32 AOT-compiled for Qualcomm NPUs (per SoC) |
 
@@ -175,6 +175,12 @@ if __name__ == "__main__":
 python transcribe.py --wav sample.wav
 ```
 
+### Android sample app
+
+For an on-device Android demo that runs Moonshine (and other ASR models) with
+hardware acceleration, see the LiteRT
+[speech recognition sample](https://github.com/google-ai-edge/litert-samples/tree/main/samples/litert/speech_recognition).
+
 ## Performance
 
 Measured on one 5 s window of continuous speech (11 output tokens), CPU
@@ -186,30 +192,35 @@ with the CPU accelerator at default threading:
 | Device | Variant | Encode | Decode | Window total | RTF |
 |---|---|---|---|---|---|
 | iPhone 17 Pro | f32 | 10.9 ms | 70.2 ms | 81.2 ms | 0.016 |
-| iPhone 17 Pro | i8 | 16.3 ms | 67.5 ms | 84.1 ms | 0.017 |
+| iPhone 17 Pro | i8 | 10.7 ms | 69.3 ms | 80.0 ms | 0.016 |
 | Apple M4 Max (macOS) | f32 | 8.1 ms | 79.4 ms | 87.5 ms | 0.017 |
-| Apple M4 Max (macOS) | i8 | 8.8 ms | 76.7 ms | 85.5 ms | 0.017 |
+| Apple M4 Max (macOS) | i8 | 7.4 ms | 72.6 ms | 80.0 ms | 0.016 |
 | Raspberry Pi 5 | f32 | 50.7 ms | 444.7 ms | 495.3 ms | 0.099 |
-| Raspberry Pi 5 | i8 | 33.4 ms | 269.4 ms | 303.1 ms | 0.061 |
+| Raspberry Pi 5 | i8 | 50.3 ms | 267.4 ms | 317.7 ms | 0.064 |
 
 RTF = processing time / audio duration (lower is better; below 1.0 is faster
 than real time). Decode dominates and scales with the number of emitted
 tokens, so dense speech takes proportionally longer than sparse speech. The
-i8 model runs about 1.6x faster than f32 on the Pi 5's Cortex-A76; on Apple
-silicon (M4 Max, iPhone 17 Pro) the two are equally fast. The f32 greedy
-decode is deterministic across platforms: the same window produces
-bit-identical token sequences on all three devices.
+i8 model's int8 decoder makes it about 1.6x faster than f32 on the Pi 5's
+Cortex-A76; on Apple silicon (M4 Max, iPhone 17 Pro) the two are equally
+fast. The greedy decode is deterministic across platforms: the same window
+produces bit-identical f32 token sequences on all three devices, the i8
+model reproduces the f32 token sequence exactly on the dense test window on
+both Apple devices, and its transcripts are identical between the M4 Max
+and the Pi 5 on all 12 test clips.
 
 ### Accuracy note
 
 In a 12-clip spot check (LibriSpeech dev-clean samples plus two
 public-domain clips), the f32 model transcribes clips of up to 5 s at
-near-reference quality. The i8 variant currently shows significant
-transcription degradation on the same clips, which isolates to its quantized
-encoder (an f32 encoder with the i8 decoder matches full-f32 output almost
-exactly). Until a recalibrated i8 encoder is published, the f32 model — or
-the f32 encoder combined with the i8 decoder — is recommended where
-transcription quality matters.
+near-reference quality, and the i8 model matches it: the same overall word
+error rate on the 12-clip harness (within chunking noise), word-level
+divergence from the f32 transcripts of 2.5%, and encoder output that is
+bit-identical to f32 because the encoder is not quantized. The encoder is kept in float32
+deliberately — the convolutional audio frontend on the raw waveform does not
+survive dynamic-range quantization (an earlier fully-quantized i8 upload
+degraded badly for exactly this reason) — while the decoder, which dominates
+latency, carries the int8 weights.
 
 For the source model's quality, the
 [Moonshine paper](https://arxiv.org/abs/2410.15608) reports that Moonshine

moonshine_tiny_5s_i8.tflite

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f45441f5710d745afdc376d0c60abe223e3daf1ad10b5327170dd40cc5fdfb3b
-size 29135952
+oid sha256:97abdeea122d579229091659c24c59d988c6419d453a200f6471241a53b9a9b9
+size 51936896