docs/REPORT_LFM_MODEL.md

# MomsVoiceAI — Fix & Optimization Report

**Date:** 2026-06-14  
**Branch:** `fix/voice-clone-card`, `fix/lfm-model-param`

---

## 1. LFM2.5-Audio Inference (`inference_lfm.py`)

### 1.1 System Prompt Fix
**Before:** Story context was embedded in the system prompt as a custom "friendly storyteller" instruction.  
**After:** System prompt simplified to the official interleaved speech-to-speech directive:
```
"Respond with interleaved text and audio."
```
Story context moved to the **user turn** as a text prefix, which is the correct placement for the LFM2.5-Audio interleaved model.

### 1.2 Audio Decode — Streaming → Post-generation
**Before:** Mimi streaming decode ran frame-by-frame during `generate_interleaved`, requiring `mimi.streaming(1)` context and manual state resets (`_reset_mimi_streaming_state`).  
**After:** Audio tokens are collected during generation, then decoded in one call via `processor.decode(audio_codes)`. This is more stable and avoids Mimi internal buffer corruption between calls.

```python
# Before (streaming, fragile)
with mimi.streaming(1):
    for t in model.generate_interleaved(...):
        if t.numel() == 8:
            wav_chunk = mimi.decode(codes)

# After (post-generation, stable)
for t in model.generate_interleaved(...):
    if t.numel() == 8:
        audio_out.append(t)
audio_codes = torch.stack(audio_out[:-1], dim=1).unsqueeze(0)  # drop EOS frame
waveform = processor.decode(audio_codes).cpu().float()[0].numpy()
```

### 1.3 Removed `torch.compile`
`torch.compile(mode="reduce-overhead", dynamic=True)` was applied to the LFM model on CUDA. This caused silent audio output and Mimi state bugs in practice. Removed in favor of eager mode.

### 1.4 Mimi Warmup in `get_lfm_model()`
An eager Mimi decode warmup now runs immediately after model load on CUDA to force lazy construction and catch device errors early:
```python
with torch.no_grad(), mimi.streaming(1):
    mimi.decode(torch.randint(0, 2048, (1, 8, 1), device=device))
```

### 1.5 `max_new_tokens` 100 → 512 → 150
**Before (original):** Default was `100` tokens — too short for a complete spoken answer.  
**After (first fix):** Raised to `512` to give the model room to finish. See §5 for why this was later revised.  
**After (current):** Reduced to `150`. See §5 for full reasoning.

### 1.6 New Functions Added
| Function | Purpose |
|---|---|
| `get_model_status()` | Returns device, dtype, GPU memory usage for diagnostics |
| `text_to_audio_lfm()` | LFM-based TTS mode (read text aloud via interleaved generation) |
| `_assemble_waveform()` | Shared waveform assembly + peak normalization helper |

---

## 2. Q&A Recording Flow (`app.py`)

### 2.1 Removed JS Auto-click on Ask Button
**Before:** Clicking "❓ Ask a Question" ran JavaScript to auto-click the Record button 100ms later. This fired before mic permission was granted, breaking browser microphone access.  
**After:** JS removed. The user clicks Record manually (required by browser security model — mic access requires a direct user gesture on the Record button itself).

### 2.2 `stop_recording` Wired Directly to Python
**Before:** `stop_recording` used `fn=None` + a JS hack that clicked the "Get Answer" submit button after a 600ms delay.  
**After:** `stop_recording` calls `auto_submit_on_stop` directly in Python, which immediately invokes LFM inference.

```
Recording stops → stop_recording → auto_submit_on_stop → build_qa_response
  → answer_question_audio (LFM inference) → answer audio plays automatically
```

### 2.3 Removed Double-trigger Race Condition
**Before:** Both `question_audio.change` and `stop_recording` (via JS) triggered submission independently, causing a race condition where LFM was called twice.  
**After:** `question_audio.change` handler removed. `stop_recording` is the single auto-trigger. `claim_audio_submission` dedup guards against any remaining duplicate calls. The "Get Answer" button remains as a manual fallback for text-only questions.

---

## 3. Voice Clone Card Management (`app.py`)

### 3.1 Persistent Voice Cards Across Page Reloads
**Before:** `voices_state` was always initialized from `mock_voices` (hardcoded placeholders). Cloned voice profiles saved to `Voice_Profile/` never appeared as cards after a page reload.  
**After:** On startup, `list_saved_profiles()` reads all saved `.pt` profiles from disk and builds `_initial_voices` to populate `voices_state`. Previously cloned voices appear as cards immediately on next launch.

### 3.2 Voice Resets to Vivian on Page Refresh
**Before:** `voice_profile_state` was initialized to `_default_profile_id` (most recently saved clone), so refreshing the browser silently loaded the clone.  
**After:** `voice_profile_state = gr.State(None)` — always starts with the stock Vivian voice. The user explicitly selects a clone by clicking its card.

### 3.3 Active Narrator Indicator
Added `narrator_info` HTML component in the Library player panel displaying the currently active voice. Updates on:
- Voice card click
- Book selection

### 3.4 Removed Fake Placeholder Voice Cards
`Grandpa Joseph` and `Aunt Sarah` were placeholder cards with no real voice profile. Clicking them silently fell through to the stock Vivian voice, which was misleading. Removed. Only real profiles and `Mom's Voice` (Vivian stock) remain.

---

## 4. Pipeline Comparison — Demo vs Production

The full speech-to-speech pipeline matches `app_qa_flow_demo.py` and `inference_lfm_fix.py`:

| Stage | Demo / Fix | Production (current) |
|---|---|---|
| Audio input SR | `librosa.load(sr=16000)` | Same ✓ |
| System prompt | `"Respond with interleaved text and audio."` | Same ✓ |
| Story context | User turn prefix | Same ✓ |
| `max_new_tokens` | `512` | `150` (reduced — see §5) |
| `audio_temperature` | `1.0` | Same ✓ |
| `audio_top_k` | `4` | Same ✓ |
| Audio decode | `processor.decode(audio_codes)` | Same ✓ |
| Normalization | `* (0.9 / peak)` | Same ✓ |
| Return values | 4 `(text, wav, sr, gen_stats)` | 3 `(text, wav, sr)` — `gen_stats` dropped for `qa_flow.py` compatibility |
| `stop_recording` trigger | Direct Python call | Same ✓ |
| Dedup | `claim_audio_submission` | Same ✓ |

The only intentional difference: `inference_lfm_fix.py` returns a 4th `gen_stats` dict. Dropped in production because `qa_flow.py` unpacks exactly 3 values (`answer_text, waveform, sr = answer_fn(...)`). Stats are logged internally instead.

---

## 5. Q&A Answer Quality — Prompt Engineering & Token Budget

### 5.1 Problem: Long, Repetitive Answers

With `max_new_tokens=512` and a minimal system prompt, the model frequently:
- Repeated the question before answering
- Produced 10–20 second audio responses for simple factual questions
- Ran out of generation budget mid-sentence on longer story contexts due to KV cache pressure

### 5.2 How `generate_interleaved` Counts Tokens

`generate_interleaved` yields one logical step at a time:
- 1 text token → `t.numel() == 1`
- 1 audio frame (8 Mimi codec tokens) → `t.numel() == 8`

`max_new_tokens` counts **logical steps**, not raw codec tokens. A 1-2 sentence spoken answer requires approximately:
- ~30 text tokens
- ~50 audio frames (~4 seconds at 12.5 fps)
- **Total: ~80 steps**

`max_new_tokens=512` allowed up to ~41 seconds of audio — 6× more than needed. If the model missed the EOS it would generate until the cap, wasting GPU memory and time.

### 5.3 Fix: `max_new_tokens` 512 → 150

150 steps covers a 4–6 second spoken answer with headroom. `text_to_audio_lfm` (TTS) retains `max_new_tokens=1024` since reading longer text aloud legitimately requires more frames.

### 5.4 Fix: Three-layer Prompt Constraint

Brevity is now enforced at three prompt positions so the model cannot ignore any single one:

**Layer 1 — System prompt** (sets global behavior):
```
Before: "Respond with interleaved text and audio."
After:  "Respond with interleaved text and audio. Give a short, direct answer in 1-2 sentences. Do not repeat the question."
```
`"Do not repeat the question"` targets the most common failure mode: the model echoing the question before answering, which consumes ~20 tokens and audio frames with no value.

**Layer 2 — User context prefix** (constrains to story content):
```
Before: f"Story context:\n{story_context[:3000]}\n\nAnswer the question in 1-2 short sentences based only on the story."
After:  f"Story context:\n{story_context[:2000]}\n\nBased only on the story above, answer briefly."
```
Context truncation also tightened from 3000 → 2000 characters (~750 → 500 tokens), leaving more of the context window budget for generation.

**Layer 3 — User turn closing** (highest recency — last text model reads before generating):
```python
chat.add_text("Answer in 1-2 sentences only.")
chat.end_turn()
```
Placed after the question (audio or text) so it is the final input before generation begins. Recency gives this the most direct influence on output length.

### 5.5 Why Three Layers?

Interleaved audio+text models do not respond to a single brevity instruction as reliably as pure text LLMs. The system prompt sets the prior, the context prefix reinforces it mid-conversation, and the closing instruction overrides drift just before generation starts. All three are needed for consistent short answers across different question types.

### 5.6 Summary of Changes in `answer_question_audio`

| Parameter | Before | After |
|---|---|---|
| `max_new_tokens` default | `512` | `150` |
| System prompt | `"Respond with interleaved text and audio."` | + brevity + no-repeat instruction |
| Story context truncation | `[:3000]` chars | `[:2000]` chars |
| Context instruction | `"Answer the question in 1-2 short sentences…"` | `"…answer briefly."` |
| Closing instruction | None | `"Answer in 1-2 sentences only."` after question |