PHASE3_PLAN.md

# Phase 3 Plan — Zero-GPU engineering wave

**Status:** planejado, não iniciado
**Date drafted:** 2026-04-27 (revised)
**Pre-req:** Phase 2 SFT done (WER 0.5316 → 0.1537, baseline → SFT, commit `79e9489`)
**Source of truth:** [`IARATTS_ROADMAP.md`](./IARATTS_ROADMAP.md) Revision 4
**Total budget:** $0 GPU, 1 wall-week, single engineer

> **Note:** the original Phase 3 (instruction+tag SFT) was renamed to Phase 4.4 in roadmap Revision 2 — it requires a multilingual continued-pretrain (Phase 4.2) as prerequisite, so it moved later. The current Phase 3 is all zero-GPU engineering wins that ship before any backbone retraining.

---

## Goal

Address ~60–70% of user-visible MOSS-Nano weaknesses via wrapper / frontend / cache code, no architecture change, no GPU spend. Foundation for Phase 4 (continued pretrain) and Phase 5 (codec swap + streaming distillation).

---

## Sub-phases

### Phase 3.1 — pt-BR text frontend (W1)

**Recommended path:** Adopt **Gruut + `num2words(lang='pt_BR')` + small custom Brazil rules** — exactly what `Piper-pt_BR-edresson` already ships. End-to-end pt-BR support: G2P, number normalization, abbreviation expansion.

**Alternative:** Author pt-BR FSTs in WeTextProcessing-engine style (~50–100 rules). `wenet-e2e/WeTextProcessing` officially supports only zh/en/ja, so this is net-new authoring.

**Patches in MOSS-Nano source:**
- Replace `resolve_text_normalization_language()` to route by **voice → language**, not by character set heuristic.
- Insert pt-BR pipeline branch between text input and tokenizer.

**Validation:** Amazon Polly pt-BR phoneme table is the gold standard.

**Effort:** S+ (2–4 days). **GPU:** $0.

---

### Phase 3.2 — Repetition-Aware Sampling (RAS, W3-decode)

**Top fix:** Lift RAS from VALL-E 2 (arXiv:2406.05370). Implementation already shipping in `FunAudioLLM/CosyVoice/cosyvoice/llm/llm.py` — copy-paste the function.

**Mechanism:** at each step compute repetition ratio of candidate token over a sliding window; if exceeds threshold τ, redraw from original distribution (instead of top-p restricted).

**Effect:** breaks infinite-loop pathology (issue #44 — same input loops 8× one seed, 200× another).

**Effort:** S (1 day). **GPU:** $0.

---

### Phase 3.3 — Sentence chunking + prompt re-injection (W3-decode + W4)

**Inference wrapper:**
- Cap `max_length` to 512 frames (~40s) per chunk.
- Re-inject the speaker prompt at every chunk boundary so AR doesn't drift.
- Hides ~80% of long-form failures pre-SFT.

**Effort:** S (1–2 days). **GPU:** $0.

---

### Phase 3.4 — Voice profile cache + reference normalization (W4)

**Layer:** Silero VAD + WavLM-base + IndexedDB.

**Mechanism:**
1. **Speaker-encoder cache:** WavLM-base (94M) runs once per voice, not per inference; cached 192-d embedding goes to IndexedDB keyed by file hash.
2. **Reference normalization:** trim leading/trailing silence (Silero VAD), peak-normalize to -23 LUFS, resample to model's native rate.
3. **API:** `cloneVoice(refWav) → voiceId` and `tts(text, voiceId)`.

**Fixes issue-#9 ("MP3 doesn't work / 6s-30s all garbage") by enforcing 3–10s clean window.**

**Effort:** S (3–4 days). **GPU:** $0.

---

### Phase 3.5 — Bug fixes (RoPE NaN + ONNX truncation)

- **Issue #28:** RoPE `inv_freq` corruption when loading via `from_pretrained` causes NaN logits. Apply upstream patch or workaround.
- **Issue #32:** ONNX path drops trailing audio. Add stop-token explicit padding before ONNX export.

**Effort:** S (1 day). **GPU:** $0.

---

### Phase 3.6 — Meta Quest 15-viseme stream emission ⭐ NEW

**Goal:** alongside the audio stream, emit a viseme stream compatible with **Meta Quest avatar SDK** (15-viseme Oculus inventory: `sil, PP, FF, TH, DD, kk, CH, SS, nn, RR, aa, E, ih, oh, ou`). **Per-viseme `start_ms`/`end_ms` timing is the priority** — accept +5–15 ms TTFT cost for tight avatar lipsync.

**Architecture:**
1. **IPA→Oculus 15-viseme lookup table (pt-BR)** — full table in [`IARATTS_VISEMES_CONTINUITY.md`](./IARATTS_VISEMES_CONTINUITY.md).
2. **Source timing:** AR LM token timestamps × phoneme duration model (Gruut/eSpeak-NG) → start/end per viseme.
3. **Output schema (HeadTTS-compatible JSON):**
   ```json
   {
     "visemes":      ["sil", "aa", "kk", "aa", "sil"],
     "vtimes_ms":    [0, 80, 160, 240, 380],
     "vdurations_ms":[80, 80, 80, 140, 60],
     "phonemes":     ["_", "a", "k", "a", "_"]
   }
   ```
4. **Optional:** 60 fps soft-weight stream `float[15]` per frame for Avatar SDK 2 / Movement SDK direct consumption.

**Reference impl:** [`met4citizen/HeadTTS`](https://github.com/met4citizen/HeadTTS) — Kokoro browser w/ visemes.

**Effort:** S+ (2–3 days). **GPU:** $0.

---

### Phase 3.7 — In-session style continuity hybrid ⭐ NEW

**Goal:** successive utterances in a session sound like the same person continuing to speak, not separate generations stitched together.

**3-layer cache (total <5 KB persistent per voice):**

| Layer | Content | Persistence | Size |
|---|---|---|---|
| **A — Speaker** | Spark-TTS BiCodec 32 tokens + StyleTTS-2 256-d style vector + WavLM 192-d | IndexedDB **forever** | ~1.5 KB |
| **B — Emotion + Prosody EMA** | rate, pitch_mean, pitch_std, energy + IndexTTS2 256-d emotion vector | sessionStorage | ~1 KB |
| **C — Audio-token tail** | Last 1.5 s = ~75 X-Codec 2 tokens, re-injected as F5-TTS-style prompt prefix at next utterance | rolling per-utterance | ~120 B |

**Inference flow:**
```
session start: load Layer A (IndexedDB) + Layer B (sessionStorage if exists, else neutral)

utterance N:
  text + Layer A speaker_emb + Layer B emotion_EMA + Layer C audio_tail (1.5s previous)
  ↓ AR LM streaming (Phase 3.2 + 3.3)
  ↓ post: extract last 1.5s tokens → Layer C update
  ↓ post: update Layer B EMA with measured rate/pitch_mean of utterance N

utterance N+1: same person continuing, no "stitched" feel
```

**Reference patterns:** F5-TTS reference encoding (Layer C); Spark-TTS BiCodec (Layer A); ElevenLabs/OpenAI session-continuity behavior (production reference).

**Effort:** S+ (3–4 days). **GPU:** $0.

---

## Acceptance criteria (Phase 3 exit)

| Test | Target | Source |
|---|---|---|
| pt-BR sentence frontend correctness (200 prompts: numbers, abbrev, dates, currency) | <5% WER vs ground-truth (Whisper-large-v3 round-trip) | Phase 3.1 |
| Infinite-loop bug repro (issue #44) | 0% loops in 50 attempts × 5 seeds each | Phase 3.2 |
| Long-form drift on 50 paragraph-length prompts | <10% drop-rate (vs ~80% baseline) | Phase 3.3 |
| Voice clone at 3s/6s/10s/30s reference | WavLM-SV cosine ≥0.65 at 3s, ≥0.75 at 6s | Phase 3.4 |
| ONNX export round-trip | 0 trailing-audio cutoff in 100 random prompts | Phase 3.5 |
| Viseme stream timing accuracy | per-viseme start/end aligned within ±20 ms vs forced-aligner ground truth | Phase 3.6 |
| Style continuity in 5-utterance session | A/B test: ≥70% of native raters say "sounds like the same person continuing" vs no-cache baseline | Phase 3.7 |

---

## Files to create when executing

- `iaratts/frontend/pt_br_pipeline.py` — Gruut + num2words + custom rules (Phase 3.1)
- `iaratts/inference/ras_sampler.py` — RAS wrapper around AR sampling (Phase 3.2)
- `iaratts/inference/chunked_decode.py` — sentence chunking + prompt re-injection (Phase 3.3)
- `iaratts/wrapper/voice_cache.ts` — IndexedDB voice profile cache + WavLM (Phase 3.4)
- `iaratts/patches/rope_nan_fix.py`, `iaratts/patches/onnx_truncation_fix.py` (Phase 3.5)
- `iaratts/avatar/oculus_viseme_emitter.py` — IPA→Oculus 15-viseme + timing extractor (Phase 3.6)
- `iaratts/avatar/ipa_to_oculus_pt_br.json` — pt-BR mapping table (Phase 3.6)
- `iaratts/wrapper/style_continuity.ts` — 3-layer hybrid cache (Phase 3.7)
- `iaratts/eval/phase3_acceptance.py` — automated acceptance suite

---

## What comes after Phase 3

Per [`IARATTS_ROADMAP.md`](./IARATTS_ROADMAP.md):

- **Phase 4** ($60–110 GPU): 48k→24k vocoder retrain, continued pt-BR+EN bilingual pretrain, hybrid TF + EOS sub-loss SFT, **paralinguistic tag SFT (Bark recipe) + IndexTTS2 instruction LM** (this is what the original "Phase 3 plan" was).
- **Phase 5.x** ($80–150 GPU): X-Codec 2 codec swap (TTFT 20ms first-frame floor), DCAR chunk-AR, Spark-TTS attribute tokens.
- **Phase 5.5 + 5.5a/b/c** ($115–190 GPU): CosyVoice-2-style streaming AR distillation to 150M, Speech Speculative Decoding (1.4×), Multi-Token Prediction 8 heads + Viterbi (4–5×), SpeakStream interleaved text-speech training. **Target TTFT: 80–180 ms WebGPU M1.**
- **Phase 6** (last resort, $80–150): FM pivot if Phase 5.5 plateaus. Streaming lost.
- **Phase 7** (polish, $20–30): OpenVoice-v2 tone-color converter; Mamba/SSM/RWKV TTS if browser ONNX matures.

---

## Companion documents

- [`IARATTS_ROADMAP.md`](./IARATTS_ROADMAP.md) — full roadmap, Revision 4
- [`IARATTS_ROADMAP_VALIDATION.md`](./IARATTS_ROADMAP_VALIDATION.md) — independent EN+ZH validation
- [`IARATTS_TTFT_VALIDATION.md`](./IARATTS_TTFT_VALIDATION.md) — TTFT optimization research
- [`IARATTS_VISEMES_CONTINUITY.md`](./IARATTS_VISEMES_CONTINUITY.md) — visemes Meta Quest + 3-layer continuity
- [`MOSS_NANO_WEAKNESSES.md`](./MOSS_NANO_WEAKNESSES.md) — source weaknesses analysis
- [`PHASE2_SFT.md`](./PHASE2_SFT.md) — Phase 2 SFT report (WER -71%)