docs/streaming-arch-design.md

Streaming zh-TW + English TTS on Jetson Nano gen-1 — Architecture Design

Status: proposal (design only; no code/training in this doc) Baseline: PrimeTTS v2 / v2.1 = MB-iSTFT-VITS, 34.7M, 16 kHz, whole-utterance Target device: Tegra X1 (Maxwell sm_53, 2 s kernel watchdog, no CUDA-graph replay; Cortex-A57 fp32-only CPU)

---

Executive summary (5 lines)

1. Recommended: a hybrid (d) — phrase-incremental input (split text at punctuation/prosodic breaks, reuse the existing encoder/duration/flow per phrase, no arch change) driving a streaming causal vocoder (cached-conv MB-iSTFT + PQMF/iSTFT overlap-add) for true incremental output.
2. Ship it in phases: Phase-1 MVP is pure chunked-reuse (option a) — zero retrain, reuse v2.1 weights, phrase-split + crossfade — which alone gets first-audio to ~200-320 ms for a short opening phrase.
3. The vocoder is the one high-value new component (Phase 2): convert the MB-iSTFT GAN to causal cached-conv with a 24-frame chunk / 4-frame lookahead; warm-start finetune from v2.1 GAN (no from-scratch). Flow is nearly frame-local and chunkable with a tiny left cache; the windowed encoder and deterministic duration predictor are already streaming-friendly.
4. Nano fit: fixed 24-frame chunk graphs keep every kernel in the millisecond range → far under the 2 s watchdog, bound the launch-overhead blowup, and dodge the long-sequence kernel that tripped the watchdog in whole-utterance v2. Per-chunk RTF ≈ 0.45 GPU / 0.60 CPU (< 1 → playback never starves). Reject AR frame-level (option c) — per-frame launch overhead on Maxwell is fatal.
5. Runtime: RapidSpeech.cpp ggml-CUDA for the streaming decoder (explicit ring-buffer conv/KV state + fixed-shape per-chunk graph), ORT-CPU fp32 as the portable fallback and for the g2pw/g2p_en frontend, which stays CPU and chunks at punctuation to preserve polyphone context.

---

1. Component diagram (recommended hybrid)

                       ┌─────────────────────────── streams in (LLM tokens / typed text) ──────────────────────────┐
                       │                                                                                            │
   text tokens ──▶  PHRASE CHUNKER (CPU)                                                                            │
                  · buffer until a phrase boundary: 。！？，、；: / EN . ! ? , ; : / soft cap ~10-12 chars          │
                  · never split inside a word / zh compound (keeps g2pw polyphone context intact)                  │
                  · emit phrase P0, P1, P2 … as boundaries arrive                                                   │
                       │                                                                                            │
                       ▼   (per phrase Pi)                                                                          │
   ┌───────────────────────────────────────────── PER-PHRASE ACOUSTIC (reuse v2.1, unchanged) ──────────────────┐ │
   │  g2pw (bopomofo+polyphone) + g2p_en (arpabet)  →  88-sym 3-embed (phone+tone+lang)                          │ │
   │        ▼                                                                                                    │ │
   │  Text encoder  (windowed rel-pos attn, window=4, 6L, hidden 192)   ── already local context                │ │
   │        ▼                                                                                                    │ │
   │  Duration predictor (deterministic) → length-regulate            ── naturally per-phoneme incremental      │ │
   │        ▼                                                                                                    │ │
   │  Normalizing FLOW (4× ResidualCouplingLayer + WN)                 ── frame-local; chunk w/ small L-cache    │ │
   │        ▼  latent z frames for phrase Pi (held in a frame queue)                                             │ │
   └─────────────────────────────────────────────────────────────────────────────────────────────────────────┘ │
                       │  z-frames pushed into a rolling queue                                                     │
                       ▼                                                                                            │
   ┌──────────────── STREAMING CAUSAL VOCODER (NEW; warm-start finetune of v2.1 GAN) ────────────────┐            │
   │  pull fixed 24-frame chunks (+4-frame right lookahead) from the queue                            │            │
   │  causal HiFiGAN-style upsample+ResBlocks  → cached conv-state ring buffers (per layer)           │            │
   │  PQMF synthesis (4 subbands)  → carry filter-tap tail                                            │            │
   │  per-band iSTFT (gen_istft_n_fft=16)      → overlap-add tail (n_fft-hop ≈ 1 frame)               │◀───────────┘
   │  → 24 frames of 16 kHz PCM per chunk                                                             │
   └─────────────────────────────────────────────────────────────────────────────────────────────────┘
                       │
                       ▼
   Audio ring buffer  → 20-40 ms crossfade at phrase seams (Phase-1 MVP) → speaker output (streamed)

Frame-rate anchor used throughout: MB-iSTFT-VITS at 16 kHz, hop_length 256 ⇒ 62.5 frames/s ≈ 16 ms/frame (verify against the shipped v2.1 config; all numbers scale linearly if hop differs).

---

2. Per-component streaming treatment (blocker analysis)

Component	Streaming blocker?	Treatment	Causal / chunk / lookahead / state
g2pw + g2p_en frontend	Soft. g2pw is a BERT polyphone disambiguator that wants sentence context.	Chunk at punctuation / prosodic breaks only. The boundary is the natural context edge, so no cross-boundary lookahead is needed. Never split a word/compound. Cap very long clauses at ~10-12 chars with a soft break at a word boundary (accepts minor polyphone risk to protect first-audio latency).	Chunk = one prosodic phrase. Lookahead = 0 (wait for the boundary). State = none. Runs on A57 CPU.
Text encoder (windowed rel-pos attn, window=4)	No, in phrase mode.	Whole phrase fits inside one call — attention window (4) is already local, so a phrase is ample context. For true token-streaming (Phase 3 only), use an Incremental-FastPitch-style receptive-field-constrained chunk-attention mask + fixed-size past K/V (their optimal past = 5 frames).	Phrase mode: no state. Token mode: past-K/V = 5 frames, right lookahead ≈ window/2 ≈ 2 phones.
Duration predictor + length regulation	No. Deterministic, per-phoneme.	Emit each phoneme's frames as its duration is predicted → intrinsically incremental. Runs per phrase.	Fully causal, no state, no lookahead.
Flow (4× ResidualCouplingLayer + WN)	Low. WN dilated convs have a bounded receptive field, but it is small (kernel 5, shallow) → nearly frame-local.	Phrase mode: run flow over the whole phrase at once (short → cheap). Streaming mode: run chunk-wise with a left-context cache = WN receptive field (a few frames of conv history per layer), à la CSSinger's causal posterior. Coupling layers mix per-frame, so this is well-behaved.	Left cache ≈ WN RF (~2-4 frames). No right lookahead needed if the encoder already produced the phrase.
Vocoder (MB-iSTFT GAN + PQMF + iSTFT)	Yes — but the easy, highest-value one. Conv net with limited RF → cleanly streamable.	Causal cached-conv rewrite (ring-buffer conv history per layer) + overlap-add for iSTFT and PQMF synthesis. Use CSSinger "natural padding" (pad the first chunk with actual z, not zeros) to kill the train/infer padding mismatch. Multi-band cuts per-chunk compute (each subband is 1/4 rate).	Chunk = 24 frames (≈ 384 ms audio). Right lookahead = 4 frames (≈ 64 ms). State = per-layer conv tails + PQMF filter-tap tail + iSTFT overlap tail (n_fft−hop ≈ 1 frame; gen_istft_n_fft=16 ⇒ tiny).

Net: only the vocoder requires a real arch/finetune change to unlock true intra-phrase incremental output. Everything upstream is either already local (encoder/duration) or trivially chunkable (flow), and in the MVP they all just run per phrase, unchanged.

---

3. Architecture-family decision

Option	Verdict	Why
(a) Chunked non-streaming (reuse v2.1, phrase-split, crossfade)	Adopt as MVP	Zero retrain, reuses every weight. Each phrase is short so "whole-utterance internally" is cheap and watchdog-safe. Weakness = prosody seams — mitigated by held speaker embed + crossfade + optional 1-word lookahead context.
(b) Streaming-native VITS (chunk encoder + streaming flow + streaming vocoder, retrained)	Partial adopt (target), from-scratch part rejected	CSSinger shows this works and improves quality, but it trains from scratch (500k steps). Too costly to jump to wholesale. We take only the pieces that warm-start cheaply (vocoder, then flow/encoder finetune).
(c) AR frame-level acoustic + streaming vocoder	Reject	Per-frame autoregression = tens of thousands of tiny kernel launches on a launch-overhead-bound Maxwell GPU with no CUDA-graph replay. Directly antagonistic to the Nano. Natural streaming, wrong device.
(d) Hybrid: phrase-incremental input + streaming causal vocoder	Recommend	Best latency/quality/effort trade for the Nano. Input side reuses v2.1 as-is (no retrain); output side gets one warm-start vocoder finetune. Fixed small chunks respect the watchdog and cap launch overhead. Cleanly phase-able: (a) → (d) → optional (b).

Recommendation: (d), delivered as MVP=(a) then the streaming vocoder. Rationale for the Nano specifically: it keeps the model small and fp32 (no int8/fp16 dependence — matches the A57 reality), turns the one uncapped long-sequence kernel (the whole-utterance watchdog risk) into bounded 24-frame kernels, and needs at most a vocoder finetune rather than a from-scratch retrain, so it composes with the existing v2.1 checkpoints and the RapidSpeech.cpp mbistft-vits graph we already have.

---

4. Latency + RTF budget (Nano)

Anchors: whole-utterance RTF 0.42 GPU-ggml-CUDA, 0.52 CPU-ORT fp32 @4thr; 16 ms/frame; 2 s watchdog.

Phase-1 MVP (chunked-reuse, no retrain)

First audio = frontend(P0) + whole-phrase synth(P0). Make P0 deliberately short (first 2-3 phones, ~0.25-0.5 s audio = 15-30 frames):

• g2pw on a short phrase (A57 CPU, small BERT): ~40-80 ms (Phase-0 must measure this — it is the real floor).
• synth 15-30 frames = 0.24-0.48 s audio × 0.42 GPU ≈ 100-200 ms (× 0.52 CPU ≈ 125-250 ms).
• crossfade/buffer: ~20-40 ms.
• First-audio ≈ 180-320 ms — hits the ~200-300 ms target if P0 is kept to the first few phones and grows later phrases. Longer phrases stream behind it at RTF 0.42 (< 1), so no starvation.
• Watchdog: cap phrase ≤ ~90 frames (1.44 s audio ⇒ ~600 ms compute) → safe.

Phase-2 target (streaming causal vocoder)

First audio = frontend(P0) + enough encoder/dur/flow to fill one vocoder chunk + vocode(24 frames):

• encoder/dur/flow only needs ~4-6 phones (24 + 4 lookahead frames).
• produce+vocode 28 frames = 0.45 s audio × 0.45 (chunked, +10% cache overhead) ≈ ~200 ms GPU; encoder/flow are cheaper than the vocoder so the effective figure is lower.
• First-audio ≈ 150-220 ms; steady state emits a 24-frame (384 ms) chunk every ~170 ms ⇒ per-chunk RTF ≈ 0.45 GPU / 0.60 CPU (< 1).
• Watchdog: a 24-frame chunk processes ≤ 24×256 = 6144 samples per layer → single-digit-ms kernels, zero watchdog risk, fixed shape.

Conclusion: the target ~200-300 ms first-audio is reachable already in the MVP with a short opening phrase, and comfortably so with the streaming vocoder. Per-chunk RTF stays < 1 on both runtimes.

---

5. Reuse-vs-retrain plan

Phase	Weights	Training
1 (MVP, option a)	100% reuse of v2.1 (encoder, duration, flow, MB-iSTFT GAN, speaker embeds).	None. Pure inference orchestration: phrase chunker + z/audio queues + crossfade. Hold the speaker embedding constant across chunks; optionally carry 1-word lookahead context into the next phrase to smooth prosody.
2 (streaming vocoder)	Warm-start the MB-iSTFT decoder from v2.1 GAN.	Short finetune with causal convs + fixed receptive-field chunk mask + "natural padding". Losses unchanged (mel + adversarial + feature-matching); add a 2-frame lookahead / chunk-boundary consistency loss (from streaming-VC lit) so chunk seams match the full-context output. Same corpus; add phrase-boundary segmentation metadata. Flow either kept phrase-level (no change) or finetuned with causal WN padding. Expect near-parity (Incremental FastPitch: ~8% mel-dist ↑, MOS parity; CSSinger: quality up). Gate on resynth CER, not spectra (per the aligner lesson).
3 (optional, streaming encoder/flow)	Warm-start encoder + flow.	Finetune encoder with an Incremental-FastPitch static chunk-attention mask (past=5) and causal flow. Only pursue if Phase-1/2 phrase-boundary latency proves insufficient for token-by-token LLM input.

Corpus/loss changes are minimal: no new data, just boundary metadata and one auxiliary consistency loss. The v2.1 multi-speaker path is unaffected — the speaker embedding is global and simply held constant while streaming.

---

6. Deployment path (RapidSpeech.cpp vs ORT)

Primary: RapidSpeech.cpp ggml-CUDA for the streaming decoder. It is the better streaming fit because:

• We own the graph → we can build a fixed-shape 24-frame per-chunk graph and re-run it per chunk. sm_53 has no CUDA-graph replay, but fixed shapes still eliminate per-run shape-inference/allocation churn and, crucially, bound every kernel's sequence length so nothing approaches the 2 s watchdog (this is exactly the lever that removes the long-sequence kernel which tripped the watchdog in whole-utterance v2).
• Stateful streaming is explicit: keep the ring-buffer conv tails, PQMF filter tails, iSTFT overlap tails, and (Phase 3) encoder past-K/V as persistent ggml tensors carried across chunk invocations.

Secondary / fallback: ORT-CPU fp32. Competitive for the vocoder (0.52 whole-utterance) and fully portable, but stateful streaming in ONNX means threading state tensors through graph inputs/outputs (a stateful export) — more painful than owning the ggml graph. Use ORT-CPU for the g2pw/g2p_en frontend (stays on CPU regardless) and as the portable path where CUDA isn't available.

Split of work: frontend on A57 CPU (ORT or native); acoustic + streaming vocoder on the Maxwell GPU via RapidSpeech.cpp with per-chunk state. All fp32 — no reliance on int8 dot-product or fp16 arithmetic the A57/Maxwell lack for fast paths.

---

7. Risks + open questions

1. Prosody discontinuity at phrase seams (MVP) — F0/energy jumps across chunks. Mitigate: hold speaker/prosody embedding constant, 20-40 ms crossfade, 1-word lookahead context. Open: is it audible enough (CMOS) to force the streaming vocoder sooner?
2. g2pw latency on A57 — the polyphone BERT per phrase sets the first-audio floor. Open: is it fast enough per short phrase, or does it need quantization / a lighter polyphone head / caching?
3. Cross-boundary polyphone errors — splitting near a polyphone whose disambiguating context sits across the boundary. Mitigate: split only at punctuation/strong breaks; never mid-word/compound.
4. Causal-vocoder finetune warm-start — train/infer padding mismatch. Mitigate: "natural padding" + lookahead-consistency loss. Open: does it hold at 16 kHz with PQMF, verified by resynth CER (not MCD/spectra)?
5. iSTFT/PQMF overlap-add correctness — PQMF near-perfect-reconstruction filters have boundary transients; overlap-add must be exact. Carry filter-tap and iSTFT tails; validate bit-level continuity + resynth CER.
6. Chunked flow quality — small left cache may degrade the coupling bijector at seams. Low risk (small WN RF); confirm in the Phase-2 experiment; fall back to phrase-level flow if needed.
7. Watchdog on any remaining uncapped kernel — audit that no per-chunk kernel exceeds the 24-frame bound (e.g. a global attention/normalization slipping in).

---

8. Phased build plan

• Phase 0 — measure & scaffold (no model changes). Profile on the Nano: g2pw latency per short phrase, and whole-phrase synth latency for 15/30/60/90-frame phrases (GPU + CPU). Establish the real first-audio floor. Build the phrase chunker, z/audio ring buffers, and crossfade harness. Deliverable: measured latency table + streaming harness.
• Phase 1 — MVP chunked-reuse (option a), zero retrain. Phrase splitter (punctuation + ≤90-frame cap, word-safe) → synth each phrase with v2.1 → 20-40 ms crossfade → stream. Hold speaker embed; optional 1-word lookahead context. Ship. Deliverable: streaming demo; first-audio + CMOS-vs-non-streaming numbers. This likely already meets ~200-300 ms first-audio for conversational (LLM emits clause-by-clause).
• Phase 2 — streaming causal vocoder (warm-start finetune) → hybrid (d). Causal cached-conv MB-iSTFT + PQMF/iSTFT overlap-add, chunk=24 / lookahead=4, natural padding, lookahead-consistency loss; warm-start from v2.1 GAN. Implement ring-buffer state in a fixed-24-frame RapidSpeech.cpp graph. Now true intra-phrase incremental output; first-audio ~150-220 ms; watchdog-proof. Deliverable: streaming vocoder checkpoint + resynth-CER gate + on-device RTF.
• Phase 3 — optional streaming encoder/flow (option b pieces). Chunk-attention encoder (Incremental-FastPitch static mask, past=5) + causal flow, warm-start finetune → enables token-by-token LLM input without waiting for a phrase boundary. Only if Phase-1/2 boundary latency is insufficient.

---

Sources

• Incremental FastPitch: Chunk-based High Quality Text to Speech — arXiv:2401.01755 (chunk-based FFT, receptive-field-constrained chunk attention mask, fixed-size past K/V; 30-frame chunk, optimal past=5, ~30 ms first-chunk, MOS parity with parallel).
• CSSinger: End-to-End Chunkwise Streaming SVS based on Conditional VAE — arXiv:2412.08918 (ChunkStream decoder 20 frames / 10 left / 4 right; causal HiFiGAN + "natural padding"; causal posterior; trains from scratch; quality ≥ parallel; CPU RTF 0.635).
• Comparative Analysis of Fast and High-Fidelity Neural Vocoders for Low-Latency Streaming in Resource-Constrained Environments — arXiv:2506.03554 (HiFiGAN / iSTFTNet / multi-band streaming; cached conv state; RTF < 1 on edge; multi-band lowers per-chunk compute).
• VoXtream: Full-Stream TTS with Extremely Low Latency — arXiv:2509.15969 (102 ms first-packet, output starts after first word).
• SpeakStream: Streaming TTS with Interleaved Data — arXiv:2505.19206 (decoder-only, 45 ms first-token, interleaved speech-text).
• SyncSpeech: Low-Latency Dual-Stream TTS with Temporal Masked Transformer — arXiv:2502.11094 (chunk-aware decoder).
• Streaming Voice Conversion through Chunk-wise Training and Lookahead Loss — Univ. Rochester ECE477 2024 (2-frame lookahead loss for chunk consistency).
• S5-TTS / Streaming T5-based TTS with Limited Lookahead — arXiv:2606.21882 (lookahead-causal masking, word-by-word).
• NVIDIA Riva TTS streaming docs — chunked FastPitch + HiFi-GAN, time-to-first-audio streaming (docs.nvidia.com Riva TTS).