Inflect-Nano-v2 — LiteRT, dynamic length + exact streaming

Inflect-Nano-v2 (4.0M params, VITS-family end-to-end TTS, English, fixed male voice, 24 kHz, Apache-2.0) converted to LiteRT CPU/XNNPACK graphs with a dynamic sequence length and exact intra-sentence streaming (overlap-discard chunking reproduces the full decode at corr 1.000000). Built for small-CPU targets (Raspberry Pi class): RTF 0.111 measured on a Raspberry Pi 5, time-to-first-audio ~220 ms — Piper-class speed.

Listen (golden sentence, same inputs/noise): samples/litert_golden.wav — this port (fp32) · samples/ref_torch.wav — the PyTorch reference (waveform corr 1.000000).

The upstream repo ships a PyTorch checkpoint + runtime. This port re-authors the VITS inference graph in TF (weights loaded from model.pth) and converts with the official TFLiteConverter, keeping both sequence axes dynamic.

Graphs

Graph	Inputs	Outputs	fp32	fp16
inflect_text_encoder.tflite	tokens [1,N] int32	m_p [1,N,128], logs_p [1,N,128], logw [1,N,1]	3.5 MB	1.8 MB
inflect_decoder.tflite	z_p [1,T,128]	wav [1,256·T] @ 24 kHz	12.6 MB	6.4 MB

Host glue: durations = ceil(exp(logw)/speed); expand m_p/logs_p with np.repeat; z_p = m_p + randn·exp(logs_p)·variation (noise generated host-side for reproducibility); decoder → waveform. use_sdp=false in this checkpoint, so the duration predictor is deterministic convs — no spline flows anywhere.

Measured on a real Raspberry Pi 5 (2026-08-06)

Pi 5 8 GB, Raspberry Pi OS 64-bit, Python 3.13.5, ai-edge-litert 2.1.6, 4 threads. vcgencmd get_throttled = 0x0 before/after each fp32 run.

Sentence	N tokens	audio	encoder	decoder	sentence	RTF	TTFA
[0]	75	2.07 s	2.7 ms	220.4 ms	223.1 ms	0.108	218.8 ms
[1]	121	3.21 s	4.9 ms	351.1 ms	356.0 ms	0.111	221.2 ms
[2]	269	7.19 s	22.2 ms	788.3 ms	810.5 ms	0.113	239.0 ms

Overall RTF 0.111 (fp32), streaming exact (corr 1.000000) and full-waveform ref-corr 1.000000 — Piper-class speed (Piper lessac-low baseline on this device class: RTF 0.10, 147 ms/phrase). ⚠ fp16 is speed-identical but showed a real quality break on one bench sentence (ref-corr 0.388) — the flow layers are fp16-sensitive; deploy fp32.

GPU (v3dv WebGPU) status — measured on the Pi 5, 2026-08-06

With Mesa built from git (v3dv Vulkan 1.3, driver 26.2.99, V3D_WEBGPU_OVERRIDE=1): the static-chunk decoder (inflect_decoder_static228.tflite) compiles and runs fully accelerated (is_fully_accelerated=True) with output corr 0.9904 vs CPU (fp16-class divergence; the flow layers are precision-sensitive — listen before adopting). Dynamic graphs do not compile (static shapes required). CPU remains the recommended deployment (faster on this board).

Verification (vs. PyTorch reference, same inputs/noise)

Check	Result
text encoder (m_p / logs_p / logw)	maxerr ≤ 2.3e-6
decoder wav, golden sentence	corr 1.000000, maxerr 2.6e-5
dynamic lengths	N = 49 / 165 / 217, T = 134 / 430 / 527 on the same graphs
streaming vs full decode	corr 1.000000 (maxerr ≤ 1e-6)
fp16 decoder	corr 0.999879 (but see the fp16 warning above)

Speed (Mac M-series, 4 threads, XNNPACK)

Sentence	N	T	audio	encoder	decoder	RTF
golden	165	430	4.59 s	3 ms	88 ms	0.020
short	49	134	1.43 s	1 ms	28 ms	0.020
long	217	527	5.62 s	3 ms	99 ms	0.018

In a python:3.12-slim linux/arm64 container (same aarch64 ai-edge-litert 2.1.6 wheel the Pi uses): waveform corr 1.000000 vs the Mac output, streaming corr 1.000000.

Snapdragon NPU (Hexagon)

• inflect_decoder.tflite — the NPU runs it at 121.2 ms. The GPU does not — LiteRtException: Failed to compile model.
• inflect_decoder_fp16.tflite — the NPU runs it at 131.7 ms. The GPU does not — LiteRtException: Failed to compile model.
• inflect_decoder_static228.tflite — the GPU is faster: 8.73 ms against 75.29 ms on the NPU, a factor of 8.63. The NPU still loads 12.51x faster (152 ms against 1896 ms).
• inflect_text_encoder.tflite — the NPU runs it at 24.82 ms. The GPU does not — LiteRtException: Failed to compile model.
• inflect_text_encoder_fp16.tflite — the NPU runs it at 26.34 ms. The GPU does not — LiteRtException: Failed to compile model.

file	backend	compiled	inference (median / min)	load
inflect_decoder.tflite	NPU (Hexagon v81)	on-device JIT	121.2 ms / 119.0 ms	538 ms
inflect_decoder_fp16.tflite	NPU (Hexagon v81)	on-device JIT	131.7 ms / 130.8 ms	594 ms
inflect_decoder_static228.tflite	NPU (Hexagon v81)	on-device JIT	75.29 ms / 72.71 ms	152 ms
inflect_decoder_static228.tflite	GPU (Adreno)	—	8.73 ms / 8.66 ms	1896 ms
inflect_text_encoder.tflite	NPU (Hexagon v81)	on-device JIT	24.82 ms / 24.52 ms	246 ms
inflect_text_encoder_fp16.tflite	NPU (Hexagon v81)	on-device JIT	26.34 ms / 26.10 ms	251 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.68–0.81, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 989 ms to 27 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Quickstart

hf download litert-community/Inflect-Nano-v2 --local-dir inflect-litert
cd inflect-litert

One-command benchmark (no espeak needed on the device — inputs are pre-tokenized in bench_inputs.npz):

pip install numpy ai-edge-litert
python bench.py --models-dir .                # fp32, 4 threads
python bench.py --models-dir . --precision fp16 --write-wavs

Reports encoder/decoder latency, RTF, streaming time-to-first-audio, and waveform-correlation identity checks (full decode vs the bundled reference, streamed vs full).

Drop-in synthesis (Piper replacement; mirrors the upstream inference.py sentence handling — pauses, edge-fade, per-sentence seed):

pip install numpy ai-edge-litert phonemizer espeakng-loader num2words Unidecode
python say.py "Hello! How can I help you today?" --models-dir . --frontend-dir frontend -o hello.wav

from say import InflectTTS

tts = InflectTTS(models_dir=".", frontend_dir="frontend")
for sentence, pcm in tts.stream(text):   # float32 @ 24 kHz per sentence
    play(pcm)

frontend/ contains the upstream Apache-2.0 text frontend (owensong/Inflect-Nano-v2); it phonemizes with espeak-ng (GPL-3.0, in-process — same situation as Piper's frontend).

Streaming (exact)

The decoder (flow + HiFi-GAN generator) is fully convolutional with no normalization layers, so overlap-discard chunking is exact: 100-frame chunks (+64 frames context each side) reproduce the full decode at corr 1.000000. Time-to-first-audio is encoder + one chunk (~220 ms on the Pi 5). This is the model to use when true sub-sentence streaming matters (compare: KittenTTS's AdaIN statistics make its chunked mode approximate).

Files

File	Purpose
inflect_{text_encoder,decoder}.tflite	fp32 graphs (recommended)
inflect_{text_encoder,decoder}_fp16.tflite	fp16-weight variants (⚠ flow layers are fp16-sensitive — deploy fp32)
inflect_decoder_static228.tflite	static 228-frame decoder chunk (CPU recommended on the Pi 5; the only file a phone GPU compiles — 8.73 ms on a Galaxy S26 Adreno, see Snapdragon NPU (Hexagon))
frontend/	upstream Apache-2.0 text frontend (phonemization + cleaners)
bench.py + bench_inputs.npz	one-command device benchmark (numpy + ai-edge-litert only)
make_bench_inputs.py	regenerate bench inputs (needs espeak on the host)
say.py	drop-in say(text) / stream(text) synthesis module + CLI
samples/	output samples: this port vs the PyTorch reference, same inputs

Conversion notes

• litert-torch dynamic export is a dead end (0.9.2): beyond the known dynamic-LSTM wall, even a plain conv stack exported with torch.export.Dim bakes the trace length into internal RESHAPEs and fails at any other length; F.embedding doesn't lower with a symbolic axis at all. The TF/Keras → TFLiteConverter path handles all of it (shape-computed reshapes, fused dynamic LSTM).
• VITS's relative-position attention (window 4) uses pad/reshape "skew" tricks; in TF they convert fine. (Under torch.export they generate unprovable divisibility/stride guards.)
• tf_keras (Keras 2) is required: Keras 3 models leave READ_VARIABLE resource ops in the converted graph.

Text frontend / licensing

English phonemization via the upstream frontend = espeak-ng (GPL-3.0) + num2words. Run espeak as a separate process, or swap a DeepPhonemizer-based neural G2P for a GPL-free stack (reference implementation: litert-community/Kokoro-G2P-en-US; note this model uses its own symbol table, so the G2P output must be remapped). Model: Apache-2.0 (BigVGAN/VITS third-party notices in the upstream repo).