v2stream_streaming/README.md

PrimeTTS v2-Stream — token-level streaming zh-TW/English TTS

A streaming MB-iSTFT-VITS (Xinran voice, 16 kHz): band-attention encoder + causal FFN + causal MB-iSTFT vocoder, so audio can be emitted incrementally in fixed chunks. Bit-exact between whole-utterance and chunked inference.

Streaming params: chunk = 24 frames (~384 ms), left cache = 64, right lookahead = 4 (1 frame = 256 samples @ 16 kHz). First-audio ≈ enc(once) + one chunk.

This is a streaming-capable variant of PrimeTTS v2 (a distinct checkpoint, not v2 run in a streaming loop). The causal + limited-lookahead architecture trades some quality for streamability vs the non-streaming v2.

Files

file	purpose
v2stream_enc.onnx	text (x,tone,lang,x_lengths,noise_scale,length_scale) → z[1,192,T]. Run once.
v2stream_dec.onnx	z[1,192,Tc] → wav[1,1,Tc·256]. Run per chunk (overlap-save).
primetts_v2stream_xinran_f32.gguf	single-graph gguf for RapidSpeech.cpp ggml (MBISTFT_STREAM=1).
onnx_stream.py	reference streaming runner (ORT).
meta.json	params + frontend convention.

Run A — ONNX Runtime (Python, torch-free)

from onnx_stream import StreamingTTS      # in this repo
tts = StreamingTTS("v2stream_enc.onnx", "v2stream_dec.onnx")
z = tts.encode(phone_ids, tone_ids, lang_ids)   # once
for pcm_chunk in tts.stream(z):                  # per 24-frame chunk
    play(pcm_chunk)                              # 16 kHz float32

phone_ids/tone_ids/lang_ids come from the frontend (text → ids): g2pw (bopomofo, Taiwan readings) + g2p_en (arpabet), 88 syms / 6 tones / 2 langs, with add_blank (0 interleaved between tokens). See frontend_bopomofo in the demo Space.

Run B — sherpa-onnx fork (C++/ORT)

OfflineTtsMbistftStreamModel(enc=..., dec=..., num_threads=2).generate(x,tone,lang, callback=...) emits audio per chunk via the callback. Branch: vieenrose/sherpa-onnx @ mbistft-streaming-tts.

Run C — RapidSpeech.cpp (ggml, GPU/Jetson)

MBISTFT_STREAM=1 MBISTFT_STREAM_LOOKAHEAD=5 mbistft-parity primetts_v2stream_xinran_f32.gguf inputs.json out --stream-chunks. Branch: vieenrose/RapidSpeech.cpp @ jetson-nano-gen1. Validated on Jetson Nano (Tegra X1): parity cosine > 0.99, first-audio ~285 ms.

License: Apache-2.0. Part of the Luigi/PrimeTTS family.