decoder-transformer CPU placement (Mali audio quality) + clean Pixel samples: README.md

README.md

@@ -16,14 +16,14 @@ tags:
 # Pocket-TTS-LiteRT
 
 [Pocket TTS](https://huggingface.co/kyutai/pocket-tts) (Kyutai, ~100M params) converted to
-**LiteRT `CompiledModel` graphs that run fully on a phone GPU**. Four stateless graphs +
-host-side orchestration reproduce the reference `pocket_tts` pipeline; on a Snapdragon
-phone the whole thing — language model, flow head and Mimi decoder — runs on the GPU at
-several times real-time.
+**LiteRT `CompiledModel` graphs for the phone GPU**. Stateless graphs + host-side
+orchestration reproduce the reference `pocket_tts` pipeline: the 100M language model, the
+flow head and the SEANet vocoder run on the mobile GPU; the small 2-layer Mimi decoder
+transformer runs on CPU (placement notes below).
 
 ![hero](assets/hero.png)
 
-Real output from the phone (nothing cloud, nothing post-processed):
+Real output from the phone (Pixel 8a, nothing cloud, nothing post-processed):
 
 | voice | sample |
 |---|---|
@@ -42,7 +42,7 @@ draw into the next latent (LSD, 1 step); a 20M tiny Mimi (×16 ConvTranspose ups
 | `pt_flowlm_fused` | emb[1,1,1024] + cos/sin + mask[1,16,1,513] + packed KV [1,96,512,64] + noise[1,32] → [1,12321] = eos ∣ latent ∣ new-k ∣ new-v | one full AR frame (step + flow head) in one invocation with one readback — the variant the Android sample runs; on Mali the per-frame cost is dispatch/sync-bound, and fusing removes one invocation + three readbacks per frame |
 | `pt_flowlm_step` | emb[1,1,1024] + cos/sin + mask[1,16,1,513] + packed KV [1,96,512,64] → cond, eos, new k/v | one AR step; KV cache lives on the host (split reference variant) |
 | `pt_flow_head` | cond[1,1024] + noise[1,32] → latent[1,32] | LSD time embeddings (s=0, t=1) baked into the cond bias (split reference variant) |
-| `pt_mimi_dec_tx` | lat[1,65,32] → feat[1,512,1024] | Mimi decoder transformer in 64-frame blocks (32-frame overlap: the 2-layer sliding-window attention has a 498-position stacked receptive field) |
+| `pt_mimi_dec_tx` | lat[1,65,32] → feat[1,512,1024] | Mimi decoder transformer in 64-frame blocks (32-frame overlap: the 2-layer sliding-window attention has a 498-position stacked receptive field) — **runs on CPU in the shipped app** (see Placement) |
 | `pt_mimi_deconly` | feat[1,512,4096] → audio[1,1,491520] | SEANet decoder, one-shot 256-frame window (causal ⇒ exact per frame) |
 
 Host side (a few hundred lines of Kotlin/Python, no FFT anywhere): sentencepiece unigram
@@ -53,17 +53,25 @@ The `pt_voice_*.bin` files are Kyutai's published per-voice prompt states repack
 packed-KV layout (fp16). Text is chunked at ≤50 tokens along sentence boundaries, exactly
 like the reference implementation.
 
-## Measured on device
-
-LiteRT 2.1.6, fp16 graphs, every graph fully delegated (`LITERT_CL`, zero CPU fallback
-nodes), decode after generation (no streaming), app process warm:
-
-* Samsung SM-S942Q (Snapdragon SM8850, Adreno): 8.2 s of speech in 1.63 s — **5.0×
-  real-time** (42-token single chunk, alba); 13.0 s in 3.05 s — **4.3×** (3 chunks,
-  marius). Measured with the split step+head graphs.
-* Pixel 8a (Tensor G3, Mali-G715): 7.9 s in 7.7 s — **~1.0× real-time** (alba); 12.5 s in
-  13.1 s (3 chunks, marius). Measured with `pt_flowlm_fused`; the gap to Adreno is
-  per-step overhead (25 MB packed-KV upload + kernel dispatch), not arithmetic.
+## Placement and measured numbers
+
+Every graph compiles fully on the GPU (`LITERT_CL`, zero CPU-fallback nodes) on both
+devices tried. The **shipped placement still runs `pt_mimi_dec_tx` on CPU**: on a Pixel
+8a's Mali the GPU output of that one graph is audibly degraded (alba voicing HNR 0.9 dB
+on GPU vs 2.8 dB on CPU — CPU matches the fp32 desktop reference exactly), and requesting
+FP32 GPU precision does not recover it. This mirrors what the Mimi zoo module documents
+for its own decoder transformer, so the same split ships here: heavy compute (LM, flow
+head, SEANet) on GPU, the 2-layer decoder transformer on CPU. It is 7 small calls per
+utterance — on the Pixel the whole pipeline goes 1.03× → 1.01× real-time.
+
+LiteRT 2.1.6, fp16 graphs, decode after generation (no streaming), app process warm:
+
+* Pixel 8a (Tensor G3, Mali-G715), shipped placement: 8.8 s of speech in 8.75 s —
+  **~1.0× real-time** (alba); per-step cost is KV upload + dispatch overhead, not
+  arithmetic.
+* Samsung SM-S942Q (Snapdragon SM8850, Adreno), measured with the all-GPU placement and
+  the split step+head graphs: 8.2 s in 1.63 s — **5.0× real-time**; 13.0 s in 3.05 s —
+  **4.3×** (3 chunks). The decoder-transformer-on-CPU delta measured ~2% on the Pixel.
 
 Numbers move with device, thermals and text length; treat them as one measured point, not
 a benchmark.