Hugging Face's logo

marcosremar2
/
iaratts-roadmap

Model card Files
xet
 Community
iaratts-roadmap / PHASE3_PLAN.md
marcosremar2's picture
marcosremar2
Upload folder using huggingface_hub
71c511c verified
preview code
|
Raw
History Blame Contribute Delete
9.36 kB

Phase 3 Plan β€” Zero-GPU engineering wave

Status: planejado, nΓ£o iniciado Date drafted: 2026-04-27 (revised) Pre-req: Phase 2 SFT done (WER 0.5316 β†’ 0.1537, baseline β†’ SFT, commit 79e9489) Source of truth: IARATTS_ROADMAP.md Revision 4 Total budget: $0 GPU, 1 wall-week, single engineer

Note: the original Phase 3 (instruction+tag SFT) was renamed to Phase 4.4 in roadmap Revision 2 β€” it requires a multilingual continued-pretrain (Phase 4.2) as prerequisite, so it moved later. The current Phase 3 is all zero-GPU engineering wins that ship before any backbone retraining.

Goal

Address ~60–70% of user-visible MOSS-Nano weaknesses via wrapper / frontend / cache code, no architecture change, no GPU spend. Foundation for Phase 4 (continued pretrain) and Phase 5 (codec swap + streaming distillation).

Sub-phases

Phase 3.1 β€” pt-BR text frontend (W1)

Recommended path: Adopt Gruut + num2words(lang='pt_BR') + small custom Brazil rules β€” exactly what Piper-pt_BR-edresson already ships. End-to-end pt-BR support: G2P, number normalization, abbreviation expansion.

Alternative: Author pt-BR FSTs in WeTextProcessing-engine style (~50–100 rules). wenet-e2e/WeTextProcessing officially supports only zh/en/ja, so this is net-new authoring.

Patches in MOSS-Nano source:

  • Replace resolve_text_normalization_language() to route by voice β†’ language, not by character set heuristic.
  • Insert pt-BR pipeline branch between text input and tokenizer.

Validation: Amazon Polly pt-BR phoneme table is the gold standard.

Effort: S+ (2–4 days). GPU: $0.

Phase 3.2 β€” Repetition-Aware Sampling (RAS, W3-decode)

Top fix: Lift RAS from VALL-E 2 (arXiv:2406.05370). Implementation already shipping in FunAudioLLM/CosyVoice/cosyvoice/llm/llm.py β€” copy-paste the function.

Mechanism: at each step compute repetition ratio of candidate token over a sliding window; if exceeds threshold Ο„, redraw from original distribution (instead of top-p restricted).

Effect: breaks infinite-loop pathology (issue #44 β€” same input loops 8Γ— one seed, 200Γ— another).

Effort: S (1 day). GPU: $0.

Phase 3.3 β€” Sentence chunking + prompt re-injection (W3-decode + W4)

Inference wrapper:

  • Cap max_length to 512 frames (~40s) per chunk.
  • Re-inject the speaker prompt at every chunk boundary so AR doesn't drift.
  • Hides ~80% of long-form failures pre-SFT.

Effort: S (1–2 days). GPU: $0.

Phase 3.4 β€” Voice profile cache + reference normalization (W4)

Layer: Silero VAD + WavLM-base + IndexedDB.

Mechanism:

  1. Speaker-encoder cache: WavLM-base (94M) runs once per voice, not per inference; cached 192-d embedding goes to IndexedDB keyed by file hash.
  2. Reference normalization: trim leading/trailing silence (Silero VAD), peak-normalize to -23 LUFS, resample to model's native rate.
  3. API: cloneVoice(refWav) β†’ voiceId and tts(text, voiceId).

Fixes issue-#9 ("MP3 doesn't work / 6s-30s all garbage") by enforcing 3–10s clean window.

Effort: S (3–4 days). GPU: $0.

Phase 3.5 β€” Bug fixes (RoPE NaN + ONNX truncation)

  • Issue #28: RoPE inv_freq corruption when loading via from_pretrained causes NaN logits. Apply upstream patch or workaround.
  • Issue #32: ONNX path drops trailing audio. Add stop-token explicit padding before ONNX export.

Effort: S (1 day). GPU: $0.

Phase 3.6 β€” Meta Quest 15-viseme stream emission ⭐ NEW

Goal: alongside the audio stream, emit a viseme stream compatible with Meta Quest avatar SDK (15-viseme Oculus inventory: sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, ou). Per-viseme start_ms/end_ms timing is the priority β€” accept +5–15 ms TTFT cost for tight avatar lipsync.

Architecture:

  1. IPA→Oculus 15-viseme lookup table (pt-BR) — full table in IARATTS_VISEMES_CONTINUITY.md.
  2. Source timing: AR LM token timestamps Γ— phoneme duration model (Gruut/eSpeak-NG) β†’ start/end per viseme.
  3. Output schema (HeadTTS-compatible JSON):
    {
  "visemes":      ["sil", "aa", "kk", "aa", "sil"],
  "vtimes_ms":    [0, 80, 160, 240, 380],
  "vdurations_ms":[80, 80, 80, 140, 60],
  "phonemes":     ["_", "a", "k", "a", "_"]
}
  4. Optional: 60 fps soft-weight stream float[15] per frame for Avatar SDK 2 / Movement SDK direct consumption.

Reference impl: met4citizen/HeadTTS β€” Kokoro browser w/ visemes.

Effort: S+ (2–3 days). GPU: $0.

Phase 3.7 β€” In-session style continuity hybrid ⭐ NEW

Goal: successive utterances in a session sound like the same person continuing to speak, not separate generations stitched together.

3-layer cache (total <5 KB persistent per voice):

Layer Content Persistence Size
A β€” Speaker Spark-TTS BiCodec 32 tokens + StyleTTS-2 256-d style vector + WavLM 192-d IndexedDB forever ~1.5 KB
B β€” Emotion + Prosody EMA rate, pitch_mean, pitch_std, energy + IndexTTS2 256-d emotion vector sessionStorage ~1 KB
C β€” Audio-token tail Last 1.5 s = ~75 X-Codec 2 tokens, re-injected as F5-TTS-style prompt prefix at next utterance rolling per-utterance ~120 B

Inference flow:

session start: load Layer A (IndexedDB) + Layer B (sessionStorage if exists, else neutral)

utterance N:
  text + Layer A speaker_emb + Layer B emotion_EMA + Layer C audio_tail (1.5s previous)
  ↓ AR LM streaming (Phase 3.2 + 3.3)
  ↓ post: extract last 1.5s tokens β†’ Layer C update
  ↓ post: update Layer B EMA with measured rate/pitch_mean of utterance N

utterance N+1: same person continuing, no "stitched" feel

Reference patterns: F5-TTS reference encoding (Layer C); Spark-TTS BiCodec (Layer A); ElevenLabs/OpenAI session-continuity behavior (production reference).

Effort: S+ (3–4 days). GPU: $0.

Acceptance criteria (Phase 3 exit)

Test Target Source
pt-BR sentence frontend correctness (200 prompts: numbers, abbrev, dates, currency) <5% WER vs ground-truth (Whisper-large-v3 round-trip) Phase 3.1
Infinite-loop bug repro (issue #44) 0% loops in 50 attempts Γ— 5 seeds each Phase 3.2
Long-form drift on 50 paragraph-length prompts <10% drop-rate (vs ~80% baseline) Phase 3.3
Voice clone at 3s/6s/10s/30s reference WavLM-SV cosine β‰₯0.65 at 3s, β‰₯0.75 at 6s Phase 3.4
ONNX export round-trip 0 trailing-audio cutoff in 100 random prompts Phase 3.5
Viseme stream timing accuracy per-viseme start/end aligned within Β±20 ms vs forced-aligner ground truth Phase 3.6
Style continuity in 5-utterance session A/B test: β‰₯70% of native raters say "sounds like the same person continuing" vs no-cache baseline Phase 3.7

Files to create when executing

  • iaratts/frontend/pt_br_pipeline.py β€” Gruut + num2words + custom rules (Phase 3.1)
  • iaratts/inference/ras_sampler.py β€” RAS wrapper around AR sampling (Phase 3.2)
  • iaratts/inference/chunked_decode.py β€” sentence chunking + prompt re-injection (Phase 3.3)
  • iaratts/wrapper/voice_cache.ts β€” IndexedDB voice profile cache + WavLM (Phase 3.4)
  • iaratts/patches/rope_nan_fix.py, iaratts/patches/onnx_truncation_fix.py (Phase 3.5)
  • iaratts/avatar/oculus_viseme_emitter.py β€” IPAβ†’Oculus 15-viseme + timing extractor (Phase 3.6)
  • iaratts/avatar/ipa_to_oculus_pt_br.json β€” pt-BR mapping table (Phase 3.6)
  • iaratts/wrapper/style_continuity.ts β€” 3-layer hybrid cache (Phase 3.7)
  • iaratts/eval/phase3_acceptance.py β€” automated acceptance suite

What comes after Phase 3

Per IARATTS_ROADMAP.md:

  • Phase 4 ($60–110 GPU): 48kβ†’24k vocoder retrain, continued pt-BR+EN bilingual pretrain, hybrid TF + EOS sub-loss SFT, paralinguistic tag SFT (Bark recipe) + IndexTTS2 instruction LM (this is what the original "Phase 3 plan" was).
  • Phase 5.x ($80–150 GPU): X-Codec 2 codec swap (TTFT 20ms first-frame floor), DCAR chunk-AR, Spark-TTS attribute tokens.
  • Phase 5.5 + 5.5a/b/c ($115–190 GPU): CosyVoice-2-style streaming AR distillation to 150M, Speech Speculative Decoding (1.4Γ—), Multi-Token Prediction 8 heads + Viterbi (4–5Γ—), SpeakStream interleaved text-speech training. Target TTFT: 80–180 ms WebGPU M1.
  • Phase 6 (last resort, $80–150): FM pivot if Phase 5.5 plateaus. Streaming lost.
  • Phase 7 (polish, $20–30): OpenVoice-v2 tone-color converter; Mamba/SSM/RWKV TTS if browser ONNX matures.

Companion documents