README.md

---
license: apache-2.0
library_name: litert
pipeline_tag: text-to-speech
base_model: owensong/Inflect-Nano-v2
language:
  - en
tags:
  - litert
  - tflite
  - tts
  - text-to-speech
  - on-device
  - streaming
  - raspberry-pi
  - vits
  - inflect
---

# Inflect-Nano-v2 — LiteRT, dynamic length + exact streaming

[Inflect-Nano-v2](https://huggingface.co/owensong/Inflect-Nano-v2) (4.0M params,
VITS-family end-to-end TTS, English, fixed male voice, 24 kHz, Apache-2.0)
converted to **LiteRT CPU/XNNPACK** graphs with a **dynamic sequence length**
and **exact intra-sentence streaming** (overlap-discard chunking reproduces the
full decode at corr 1.000000). Built for small-CPU targets (Raspberry Pi
class): **RTF 0.111 measured on a Raspberry Pi 5**, time-to-first-audio
~220 ms — Piper-class speed.

![Inflect-Nano-v2 — LiteRT on-device output](samples/sample.png)

**Listen** (golden sentence, same inputs/noise):
[`samples/litert_golden.wav`](https://huggingface.co/litert-community/Inflect-Nano-v2/resolve/main/samples/litert_golden.wav)
— this port (fp32) ·
[`samples/ref_torch.wav`](https://huggingface.co/litert-community/Inflect-Nano-v2/resolve/main/samples/ref_torch.wav)
— the PyTorch reference (waveform corr 1.000000).

The upstream repo ships a PyTorch checkpoint + runtime. This port re-authors
the VITS inference graph in TF (weights loaded from `model.pth`) and converts
with the official `TFLiteConverter`, keeping both sequence axes dynamic.

## Graphs

| Graph | Inputs | Outputs | fp32 | fp16 |
| ----- | ------ | ------- | ---- | ---- |
| `inflect_text_encoder.tflite` | tokens [1,N] int32 | m_p [1,N,128], logs_p [1,N,128], logw [1,N,1] | 3.5 MB | 1.8 MB |
| `inflect_decoder.tflite` | z_p [1,T,128] | wav [1,256·T] @ 24 kHz | 12.6 MB | 6.4 MB |

Host glue: `durations = ceil(exp(logw)/speed)`; expand `m_p`/`logs_p` with
`np.repeat`; `z_p = m_p + randn·exp(logs_p)·variation` (noise generated
host-side for reproducibility); decoder → waveform. use_sdp=false in this
checkpoint, so the duration predictor is deterministic convs — no spline flows
anywhere.

## Measured on a real Raspberry Pi 5 (2026-08-06)

Pi 5 8 GB, Raspberry Pi OS 64-bit, Python 3.13.5, `ai-edge-litert` 2.1.6,
4 threads. `vcgencmd get_throttled` = 0x0 before/after each fp32 run.

| Sentence | N tokens | audio | encoder | decoder | sentence | RTF | TTFA |
| -------- | -------- | ----- | ------- | ------- | -------- | --- | ---- |
| [0] | 75 | 2.07 s | 2.7 ms | 220.4 ms | 223.1 ms | 0.108 | 218.8 ms |
| [1] | 121 | 3.21 s | 4.9 ms | 351.1 ms | 356.0 ms | 0.111 | 221.2 ms |
| [2] | 269 | 7.19 s | 22.2 ms | 788.3 ms | 810.5 ms | 0.113 | 239.0 ms |

Overall **RTF 0.111** (fp32), streaming exact (corr 1.000000) and full-waveform
ref-corr **1.000000** — Piper-class speed (Piper lessac-low baseline on this
device class: RTF 0.10, 147 ms/phrase). ⚠ fp16 is speed-identical but showed a
real quality break on one bench sentence (ref-corr 0.388) — the flow layers are
fp16-sensitive; **deploy fp32**.

### GPU (v3dv WebGPU) status — measured on the Pi 5, 2026-08-06

With Mesa built from git (v3dv Vulkan 1.3, driver 26.2.99,
`V3D_WEBGPU_OVERRIDE=1`): the static-chunk decoder
(`inflect_decoder_static228.tflite`) **compiles and runs fully accelerated**
(`is_fully_accelerated=True`) with output corr 0.9904 vs CPU (fp16-class
divergence; the flow layers are precision-sensitive — listen before adopting).
Dynamic graphs do not compile (static shapes required). CPU remains the
recommended deployment (faster on this board).

## Verification (vs. PyTorch reference, same inputs/noise)

| Check | Result |
| ----- | ------ |
| text encoder (m_p / logs_p / logw) | maxerr ≤ 2.3e-6 |
| decoder wav, golden sentence | **corr 1.000000**, maxerr 2.6e-5 |
| dynamic lengths | N = 49 / 165 / 217, T = 134 / 430 / 527 on the same graphs |
| streaming vs full decode | **corr 1.000000** (maxerr ≤ 1e-6) |
| fp16 decoder | corr 0.999879 (but see the fp16 warning above) |

## Speed (Mac M-series, 4 threads, XNNPACK)

| Sentence | N | T | audio | encoder | decoder | RTF |
| -------- | - | - | ----- | ------- | ------- | --- |
| golden | 165 | 430 | 4.59 s | 3 ms | 88 ms | **0.020** |
| short | 49 | 134 | 1.43 s | 1 ms | 28 ms | 0.020 |
| long | 217 | 527 | 5.62 s | 3 ms | 99 ms | 0.018 |

In a `python:3.12-slim` **linux/arm64** container (same aarch64
`ai-edge-litert` 2.1.6 wheel the Pi uses): waveform corr **1.000000** vs the
Mac output, streaming corr 1.000000.

## Quickstart

```bash
hf download litert-community/Inflect-Nano-v2 --local-dir inflect-litert
cd inflect-litert
```

**One-command benchmark** (no espeak needed on the device — inputs are
pre-tokenized in `bench_inputs.npz`):

```bash
pip install numpy ai-edge-litert
python bench.py --models-dir .                # fp32, 4 threads
python bench.py --models-dir . --precision fp16 --write-wavs
```

Reports encoder/decoder latency, RTF, streaming time-to-first-audio, and
waveform-correlation identity checks (full decode vs the bundled reference,
streamed vs full).

**Drop-in synthesis** (Piper replacement; mirrors the upstream `inference.py`
sentence handling — pauses, edge-fade, per-sentence seed):

```bash
pip install numpy ai-edge-litert phonemizer espeakng-loader num2words Unidecode
python say.py "Hello! How can I help you today?" --models-dir . --frontend-dir frontend -o hello.wav
```

```python
from say import InflectTTS

tts = InflectTTS(models_dir=".", frontend_dir="frontend")
for sentence, pcm in tts.stream(text):   # float32 @ 24 kHz per sentence
    play(pcm)
```

`frontend/` contains the upstream Apache-2.0 text frontend
([owensong/Inflect-Nano-v2](https://huggingface.co/owensong/Inflect-Nano-v2));
it phonemizes with espeak-ng (GPL-3.0, in-process — same situation as Piper's
frontend).

## Streaming (exact)

The decoder (flow + HiFi-GAN generator) is **fully convolutional with no
normalization layers**, so overlap-discard chunking is *exact*: 100-frame
chunks (+64 frames context each side) reproduce the full decode at
corr 1.000000. Time-to-first-audio is encoder + one chunk (~220 ms on the
Pi 5). This is the model to use when true sub-sentence streaming matters
(compare: KittenTTS's AdaIN statistics make its chunked mode approximate).

## Files

| File | Purpose |
| ---- | ------- |
| `inflect_{text_encoder,decoder}.tflite` | fp32 graphs (recommended) |
| `inflect_{text_encoder,decoder}_fp16.tflite` | fp16-weight variants (⚠ flow layers are fp16-sensitive — deploy fp32) |
| `inflect_decoder_static228.tflite` | static 228-frame decoder chunk (GPU-delegate experiment; CPU deployment recommended) |
| `frontend/` | upstream Apache-2.0 text frontend (phonemization + cleaners) |
| `bench.py` + `bench_inputs.npz` | one-command device benchmark (numpy + ai-edge-litert only) |
| `make_bench_inputs.py` | regenerate bench inputs (needs espeak on the host) |
| `say.py` | drop-in `say(text)` / `stream(text)` synthesis module + CLI |
| `samples/` | output samples: this port vs the PyTorch reference, same inputs |

## Conversion notes

- **litert-torch dynamic export is a dead end** (0.9.2): beyond the known
  dynamic-LSTM wall, even a plain conv stack exported with `torch.export.Dim`
  bakes the trace length into internal RESHAPEs and fails at any other length;
  `F.embedding` doesn't lower with a symbolic axis at all. The TF/Keras →
  `TFLiteConverter` path handles all of it (shape-computed reshapes, fused
  dynamic LSTM).
- VITS's relative-position attention (window 4) uses pad/reshape "skew" tricks;
  in TF they convert fine. (Under torch.export they generate unprovable
  divisibility/stride guards.)
- `tf_keras` (Keras 2) is required: Keras 3 models leave READ_VARIABLE resource
  ops in the converted graph.

## Text frontend / licensing

English phonemization via the upstream frontend = **espeak-ng (GPL-3.0)** +
num2words. Run espeak as a separate process, or swap a DeepPhonemizer-based
neural G2P for a GPL-free stack (reference implementation:
[litert-community/Kokoro-G2P-en-US](https://huggingface.co/litert-community/Kokoro-G2P-en-US);
note this model uses its own symbol table, so the G2P output must be remapped).
Model: Apache-2.0 (BigVGAN/VITS third-party notices in the upstream repo).