docs/REPORT_LFM_MODEL.md

MomsVoiceAI — Fix & Optimization Report

Date: 2026-06-14
Branch: fix/voice-clone-card, fix/lfm-model-param

---

1. LFM2.5-Audio Inference (inference_lfm.py)

1.1 System Prompt Fix

Before: Story context was embedded in the system prompt as a custom "friendly storyteller" instruction.
After: System prompt simplified to the official interleaved speech-to-speech directive:

"Respond with interleaved text and audio."

Story context moved to the user turn as a text prefix, which is the correct placement for the LFM2.5-Audio interleaved model.

1.2 Audio Decode — Streaming → Post-generation

Before: Mimi streaming decode ran frame-by-frame during generate_interleaved, requiring mimi.streaming(1) context and manual state resets (_reset_mimi_streaming_state).
After: Audio tokens are collected during generation, then decoded in one call via processor.decode(audio_codes). This is more stable and avoids Mimi internal buffer corruption between calls.

# Before (streaming, fragile)
with mimi.streaming(1):
    for t in model.generate_interleaved(...):
        if t.numel() == 8:
            wav_chunk = mimi.decode(codes)

# After (post-generation, stable)
for t in model.generate_interleaved(...):
    if t.numel() == 8:
        audio_out.append(t)
audio_codes = torch.stack(audio_out[:-1], dim=1).unsqueeze(0)  # drop EOS frame
waveform = processor.decode(audio_codes).cpu().float()[0].numpy()

1.3 Removed torch.compile

torch.compile(mode="reduce-overhead", dynamic=True) was applied to the LFM model on CUDA. This caused silent audio output and Mimi state bugs in practice. Removed in favor of eager mode.

1.4 Mimi Warmup in get_lfm_model()

An eager Mimi decode warmup now runs immediately after model load on CUDA to force lazy construction and catch device errors early:

with torch.no_grad(), mimi.streaming(1):
    mimi.decode(torch.randint(0, 2048, (1, 8, 1), device=device))

1.5 max_new_tokens 100 → 512 → 150

Before (original): Default was 100 tokens — too short for a complete spoken answer.
After (first fix): Raised to 512 to give the model room to finish. See §5 for why this was later revised.
After (current): Reduced to 150. See §5 for full reasoning.

1.6 New Functions Added

Function	Purpose
get_model_status()	Returns device, dtype, GPU memory usage for diagnostics
text_to_audio_lfm()	LFM-based TTS mode (read text aloud via interleaved generation)
_assemble_waveform()	Shared waveform assembly + peak normalization helper

---

2. Q&A Recording Flow (app.py)

2.1 Removed JS Auto-click on Ask Button

Before: Clicking "❓ Ask a Question" ran JavaScript to auto-click the Record button 100ms later. This fired before mic permission was granted, breaking browser microphone access.
After: JS removed. The user clicks Record manually (required by browser security model — mic access requires a direct user gesture on the Record button itself).

2.2 stop_recording Wired Directly to Python

Before: stop_recording used fn=None + a JS hack that clicked the "Get Answer" submit button after a 600ms delay.
After: stop_recording calls auto_submit_on_stop directly in Python, which immediately invokes LFM inference.

Recording stops → stop_recording → auto_submit_on_stop → build_qa_response
  → answer_question_audio (LFM inference) → answer audio plays automatically

2.3 Removed Double-trigger Race Condition

Before: Both question_audio.change and stop_recording (via JS) triggered submission independently, causing a race condition where LFM was called twice.
After: question_audio.change handler removed. stop_recording is the single auto-trigger. claim_audio_submission dedup guards against any remaining duplicate calls. The "Get Answer" button remains as a manual fallback for text-only questions.

---

3. Voice Clone Card Management (app.py)

3.1 Persistent Voice Cards Across Page Reloads

Before: voices_state was always initialized from mock_voices (hardcoded placeholders). Cloned voice profiles saved to Voice_Profile/ never appeared as cards after a page reload.
After: On startup, list_saved_profiles() reads all saved .pt profiles from disk and builds _initial_voices to populate voices_state. Previously cloned voices appear as cards immediately on next launch.

3.2 Voice Resets to Vivian on Page Refresh

Before: voice_profile_state was initialized to _default_profile_id (most recently saved clone), so refreshing the browser silently loaded the clone.
After: voice_profile_state = gr.State(None) — always starts with the stock Vivian voice. The user explicitly selects a clone by clicking its card.

3.3 Active Narrator Indicator

Added narrator_info HTML component in the Library player panel displaying the currently active voice. Updates on:

• Voice card click
• Book selection

3.4 Removed Fake Placeholder Voice Cards

Grandpa Joseph and Aunt Sarah were placeholder cards with no real voice profile. Clicking them silently fell through to the stock Vivian voice, which was misleading. Removed. Only real profiles and Mom's Voice (Vivian stock) remain.

---

4. Pipeline Comparison — Demo vs Production

The full speech-to-speech pipeline matches app_qa_flow_demo.py and inference_lfm_fix.py:

Stage	Demo / Fix	Production (current)
Audio input SR	librosa.load(sr=16000)	Same ✓
System prompt	"Respond with interleaved text and audio."	Same ✓
Story context	User turn prefix	Same ✓
max_new_tokens	512	150 (reduced — see §5)
audio_temperature	1.0	Same ✓
audio_top_k	4	Same ✓
Audio decode	processor.decode(audio_codes)	Same ✓
Normalization	* (0.9 / peak)	Same ✓
Return values	4 (text, wav, sr, gen_stats)	3 (text, wav, sr) — gen_stats dropped for qa_flow.py compatibility
stop_recording trigger	Direct Python call	Same ✓
Dedup	claim_audio_submission	Same ✓

The only intentional difference: inference_lfm_fix.py returns a 4th gen_stats dict. Dropped in production because qa_flow.py unpacks exactly 3 values (answer_text, waveform, sr = answer_fn(...)). Stats are logged internally instead.

---

5. Q&A Answer Quality — Prompt Engineering & Token Budget

5.1 Problem: Long, Repetitive Answers

With max_new_tokens=512 and a minimal system prompt, the model frequently:

• Repeated the question before answering
• Produced 10–20 second audio responses for simple factual questions
• Ran out of generation budget mid-sentence on longer story contexts due to KV cache pressure

5.2 How generate_interleaved Counts Tokens

generate_interleaved yields one logical step at a time:

• 1 text token → t.numel() == 1
• 1 audio frame (8 Mimi codec tokens) → t.numel() == 8

max_new_tokens counts logical steps, not raw codec tokens. A 1-2 sentence spoken answer requires approximately:

• ~30 text tokens
• 50 audio frames (4 seconds at 12.5 fps)
• Total: ~80 steps

max_new_tokens=512 allowed up to ~41 seconds of audio — 6× more than needed. If the model missed the EOS it would generate until the cap, wasting GPU memory and time.

5.3 Fix: max_new_tokens 512 → 150

150 steps covers a 4–6 second spoken answer with headroom. text_to_audio_lfm (TTS) retains max_new_tokens=1024 since reading longer text aloud legitimately requires more frames.

5.4 Fix: Three-layer Prompt Constraint

Brevity is now enforced at three prompt positions so the model cannot ignore any single one:

Layer 1 — System prompt (sets global behavior):

Before: "Respond with interleaved text and audio."
After:  "Respond with interleaved text and audio. Give a short, direct answer in 1-2 sentences. Do not repeat the question."

"Do not repeat the question" targets the most common failure mode: the model echoing the question before answering, which consumes ~20 tokens and audio frames with no value.

Layer 2 — User context prefix (constrains to story content):

Before: f"Story context:\n{story_context[:3000]}\n\nAnswer the question in 1-2 short sentences based only on the story."
After:  f"Story context:\n{story_context[:2000]}\n\nBased only on the story above, answer briefly."

Context truncation also tightened from 3000 → 2000 characters (~750 → 500 tokens), leaving more of the context window budget for generation.

Layer 3 — User turn closing (highest recency — last text model reads before generating):

chat.add_text("Answer in 1-2 sentences only.")
chat.end_turn()

Placed after the question (audio or text) so it is the final input before generation begins. Recency gives this the most direct influence on output length.

5.5 Why Three Layers?

Interleaved audio+text models do not respond to a single brevity instruction as reliably as pure text LLMs. The system prompt sets the prior, the context prefix reinforces it mid-conversation, and the closing instruction overrides drift just before generation starts. All three are needed for consistent short answers across different question types.

5.6 Summary of Changes in answer_question_audio

Parameter	Before	After
max_new_tokens default	512	150
System prompt	"Respond with interleaved text and audio."	+ brevity + no-repeat instruction
Story context truncation	[:3000] chars	[:2000] chars
Context instruction	"Answer the question in 1-2 short sentences…"	"…answer briefly."
Closing instruction	None	"Answer in 1-2 sentences only." after question