docs: update all docs to reflect Qwen3-TTS architecture

Update README.md, SPEC.md, tech_stack.md, sprint.md, and mission.md:
- Replace QWEN-TTS-0.6B references with Qwen3-TTS-1.7B-Base
- Add Supertonic as explicit fallback in model tables
- Update architecture diagram with modular file structure
- Update VRAM budget estimates for new model lineup
- Update project structure with voice_clone.py, tts.py, inference.py
- Update risk mitigations for Qwen3-TTS latency characteristics
- Update key decisions with actual model choices and module design

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

README.md

@@ -31,17 +31,18 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
 
 | Step | What Happens | Model |
 |---|---|---|
-| **Clone** | Parent records or uploads 15–30s of audio. Voice representation is computed and cached. | QWEN-TTS-0.6B |
-| **Listen** | Story plays in the cloned voice as interruptible paragraph chunks. | QWEN-TTS-0.6B |
+| **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached server-side. | Qwen3-TTS-1.7B |
+| **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to Supertonic stock voice if no clone. | Qwen3-TTS-1.7B / Supertonic |
 | **Ask** | Child taps Ask, narration pauses, child types or speaks a question. | Whisper-small (ASR, on-demand) |
-| **Answer** | A short grounded answer is generated from the story context and spoken in the cloned voice. | Qwen2.5-3B-Instruct + QWEN-TTS-0.6B |
+| **Answer** | A short grounded answer is generated from the story context and spoken in the cloned voice. | Qwen2.5-3B-Instruct + Qwen3-TTS-1.7B |
 | **Resume** | Story continues from where it left off. | Cached chunks |
 
 ## Models
 
 | Model | Role | Size |
 |---|---|---|
-| [QWEN-TTS-0.6B](https://huggingface.co/Qwen/Qwen-TTS-0.6B) | Voice cloning + TTS | 0.6B params |
+| [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
+| [Supertonic TTS](https://huggingface.co/niclas-pham/supertonic-3) | Fast stock-voice TTS fallback | ONNX model |
 | [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) | Story Q&A | 3B params (4-bit on T4) |
 | [Whisper-small](https://huggingface.co/openai/whisper-small) | Child speech-to-text | 244M params (loaded on demand) |
 
@@ -57,7 +58,7 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
 ```bash
 git clone https://github.com/MomsVoiceAI/HuggingFace_Hack.git
 cd HuggingFace_Hack
-pip install gradio transformers torch accelerate bitsandbytes soundfile numpy
+pip install -r requirements.txt
 python app.py
 # → http://localhost:7860
 ```
@@ -65,7 +66,10 @@ python app.py
 ## Project Structure
 
 ```
-├── app.py                  # Main Gradio app
+├── app.py                  # Main Gradio app (UI + wiring)
+├── voice_clone.py          # Qwen3-TTS voice cloning + profile cache
+├── tts.py                  # Unified TTS interface (Qwen3 or Supertonic)
+├── inference.py            # ASR (Whisper) + Q&A (Qwen2.5-3B-Instruct)
 ├── requirements.txt        # Python deps
 ├── stories/                # 10 cleaned public domain story texts (TTS-ready)
 ├── story_downloader/       # Story acquisition & cleaning pipeline
@@ -74,6 +78,7 @@ python app.py
 │   └── clean_stories.py         # Strips Gutenberg boilerplate for TTS
 ├── static/                 # Custom CSS/JS for Stitch-style UI
 ├── assets/covers/          # Story cover images
+├── test_modules/           # Component and integration tests
 ├── mission.md              # Product vision
 ├── sprint.md               # 2-day hackathon sprint plan
 ├── tech_stack.md            # Technical architecture

SPEC.md

@@ -20,17 +20,18 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
 
 | Step | What Happens | Model |
 |---|---|---|
-| **Clone** | Parent records or uploads 15–30s of audio. Voice representation is computed and cached. | QWEN-TTS-0.6B |
-| **Listen** | Story plays in the cloned voice as interruptible paragraph chunks. | QWEN-TTS-0.6B |
-| **Ask** | Child taps Ask, narration pauses, child types or speaks a question. | Whisper-small (ASR, on-demand) |
-| **Answer** | A short grounded answer is generated from the story context and spoken in the cloned voice. | Qwen2.5-3B-Instruct + QWEN-TTS-0.6B |
+| **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached server-side. | Qwen3-TTS-1.7B (`voice_clone.py`) |
+| **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to Supertonic stock voice if no clone. | Qwen3-TTS-1.7B / Supertonic (`tts.py`) |
+| **Ask** | Child taps Ask, narration pauses, child types or speaks a question. | Whisper-small (ASR, on-demand, `inference.py`) |
+| **Answer** | A short grounded answer is generated from the story context and spoken in the cloned voice. | Qwen2.5-3B-Instruct + Qwen3-TTS-1.7B (`inference.py` + `tts.py`) |
 | **Resume** | Story continues from where it left off. | Cached chunks |
 
 ## Models
 
 | Model | Role | Size |
 |---|---|---|
-| [QWEN-TTS-0.6B](https://huggingface.co/Qwen/Qwen-TTS-0.6B) | Voice cloning + TTS | 0.6B params |
+| [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
+| [Supertonic TTS](https://huggingface.co/niclas-pham/supertonic-3) | Fast stock-voice TTS fallback | ONNX model |
 | [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) | Story Q&A | 3B params (4-bit on T4) |
 | [Whisper-small](https://huggingface.co/openai/whisper-small) | Child speech-to-text | 244M params (loaded on demand) |
 
@@ -46,7 +47,7 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
 ```bash
 git clone https://github.com/MomsVoiceAI/HuggingFace_Hack.git
 cd HuggingFace_Hack
-pip install gradio transformers torch accelerate bitsandbytes soundfile numpy
+pip install -r requirements.txt
 python app.py
 # → http://localhost:7860
 ```
@@ -54,7 +55,10 @@ python app.py
 ## Project Structure
 
 ```
-├── app.py                  # Main Gradio app
+├── app.py                  # Main Gradio app (UI + wiring)
+├── voice_clone.py          # Qwen3-TTS voice cloning + profile cache
+├── tts.py                  # Unified TTS interface (Qwen3 or Supertonic)
+├── inference.py            # ASR (Whisper) + Q&A (Qwen2.5-3B-Instruct)
 ├── requirements.txt        # Python deps
 ├── stories/                # 10 cleaned public domain story texts (TTS-ready)
 ├── story_downloader/       # Story acquisition & cleaning pipeline

mission.md

@@ -29,9 +29,9 @@ Latency and natural interruption are part of the demo promise: the story should
 ## Hackathon Goal (2 days)
 
 Ship a working Hugging Face Space that demonstrates:
-1. **Voice cloning** — parent uploads/records a short audio clip
-2. **Interruptible story narration** — QWEN-TTS-0.6B streams paragraph-sized chunks in the cloned voice
-3. **Voice Q&A during narration** — child taps Ask, narration pauses, Qwen2.5-3B-Instruct answers from the story context, then QWEN-TTS-0.6B speaks the answer in the cloned voice
+1. **Voice cloning** — parent uploads/records a short audio clip, Qwen3-TTS extracts speaker embedding
+2. **Interruptible story narration** — Qwen3-TTS streams sentence-sized chunks in the cloned voice (Supertonic fallback for stock voice)
+3. **Voice Q&A during narration** — child taps Ask, narration pauses, Qwen2.5-3B-Instruct answers from the story context, then Qwen3-TTS speaks the answer in the cloned voice
 
 ## Target Demo User
 

sprint.md

@@ -74,11 +74,11 @@ Ship a public Hugging Face Space: parent clones voice → story streams in that
 
 | Risk | Mitigation |
 |---|---|
-| QWEN-TTS-0.6B doesn't clone well | Fall back to longer reference audio (30s+); or use CosyVoice as backup model |
-| GPU OOM with TTS + Whisper + Qwen + Gradio | Use 4-bit/8-bit Qwen loading on T4; prefer A10G for the live demo; load Whisper only on demand; use `whisper-tiny` if needed |
-| HF Space cold start too slow | Use persistent Space (not sleep); preload model in `app.py` global scope |
+| Qwen3-TTS voice quality insufficient | Fall back to longer reference audio (30s+); use Supertonic stock voice as backup |
+| GPU OOM with Qwen3-TTS + Whisper + Qwen Q&A | All models lazy-loaded on demand; 4-bit Qwen Q&A on T4; prefer A10G for live demo |
+| HF Space cold start too slow | Use persistent Space (not sleep); lazy-load models on first use rather than at startup |
+| Qwen3-TTS synthesis too slow (~30s/sentence) | Stream sentence chunks with background thread; pre-generate next chunk while current plays; cache generated audio |
 | Qwen answer latency too slow | Keep answers to 1–2 sentences; cap story prompt length; cache answers per story/question pair |
-| Story narration feels slow | Stream paragraph chunks; play the first chunk quickly; continue remaining chunks in the background; cache full narration |
 | Interrupt/resume feels brittle | Track playback state, current chunk index, queued generation jobs, and cached next chunks |
 | ASR adds too much delay | Let text questions bypass ASR; use Whisper-tiny/base or browser transcription for demo mode |
 | Live demo fails | Pre-recorded backup video ready |
@@ -89,7 +89,10 @@ Ship a public Hugging Face Space: parent clones voice → story streams in that
 
 ```
 VoiceClone/
-├── app.py                  # Main Gradio app (all logic)
+├── app.py                  # Main Gradio app (UI + wiring)
+├── voice_clone.py          # Qwen3-TTS voice cloning + profile cache
+├── tts.py                  # Unified TTS interface (Qwen3 or Supertonic)
+├── inference.py            # ASR (Whisper-small) + Q&A (Qwen2.5-3B-Instruct)
 ├── requirements.txt        # Python deps
 ├── README.md              # HF Space description
 ├── stories/                # 10 cleaned TTS-ready story texts
@@ -113,6 +116,7 @@ VoiceClone/
 │   └── favicon.png
 ├── assets/
 │   └── covers/            # Story cover images
+├── test_modules/           # Component and integration tests
 ├── mission.md             # Product vision (this hackathon)
 ├── tech_stack.md          # Technical decisions
 ├── sprint.md              # This file
@@ -123,13 +127,15 @@ VoiceClone/
 
 ## Key Decisions (Locked)
 
-- **No database** — stateless demo, voice samples held in memory/session
+- **No database** — stateless demo, voice profiles held in server-side memory cache
 - **No auth** — open access for hackathon judges
-- **Interruptible chunked streaming** — play paragraph chunks quickly, pause on Ask, answer, then resume from the saved story position
+- **Interruptible chunked streaming** — play sentence chunks, pause on Ask, answer, then resume from the saved story position
+- **Qwen3-TTS-1.7B for voice cloning** — zero-shot speaker embedding extraction from recorded audio; synthesis in cloned voice
+- **Supertonic TTS as fallback** — fast stock voice when no voice profile exists
 - **Qwen2.5-3B-Instruct for Q&A** — keeps story comprehension local while improving latency and GPU fit
 - **Short grounded Q&A** — retrieve relevant story passages, cap answers to 1–2 sentences, then synthesize audio
 - **Button-based interruption first** — tap Ask to pause narration; always-listening voice barge-in is out of scope for the hackathon
-- **Single `app.py`** — keep it simple, refactor later
+- **Modular design** — `voice_clone.py`, `tts.py`, `inference.py` separated from `app.py` for clarity
 
 ---
 

tech_stack.md

@@ -2,20 +2,22 @@
 
 ## Architecture (Hackathon-Simple)
 
-Single Python file Gradio app. Everything runs in one process on a GPU-enabled HF Space.
+Multi-module Gradio app. Everything runs in one process on a GPU-enabled HF Space.
 
 ```
-┌─────────────────────────────────────────────────┐
-│  Hugging Face Space (GPU: T4 or A10G)           │
-│                                                 │
-│  app.py (Gradio + gr.Server)                    │
-│    ├── UI: Stitch-styled tabs/cards             │
-│    ├── QWEN-TTS-0.6B (voice clone + chunks)     │
-│    ├── Whisper-small (ASR for child questions)  │
-│    └── Qwen2.5-3B-Instruct (story Q&A)          │
-│                                                 │
-│  stories/ (3–5 .txt files, public domain)       │
-└─────────────────────────────────────────────────┘
+┌──────────────────────────────────────────────────────┐
+│  Hugging Face Space (GPU: T4 or A10G)                │
+│                                                      │
+│  app.py (Gradio UI + wiring)                         │
+│    ├── voice_clone.py                                │
+│    │     └── Qwen3-TTS-1.7B (voice clone + TTS)      │
+│    ├── tts.py (unified TTS: Qwen3 or Supertonic)     │
+│    │     └── Supertonic TTS (stock voice fallback)   │
+│    ├── inference.py                                  │
+│    │     ├── Whisper-small (ASR for child questions)  │
+│    │     └── Qwen2.5-3B-Instruct (story Q&A)         │
+│    └── stories/ (10 .txt files, public domain)       │
+└──────────────────────────────────────────────────────┘
 ```
 
 No database. No external storage. No external LLM API. Stories are flat files. Audio is generated as interruptible chunks, then cached in the session for replay and resume.
@@ -24,11 +26,12 @@ No database. No external storage. No external LLM API. Stories are flat files. A
 
 | Component | Estimated VRAM | Notes |
 |---|---|---|
-| QWEN-TTS-0.6B | ~1.2 GB | Always loaded |
-| Qwen2.5-3B-Instruct (4-bit) | ~2 GB | Always loaded |
-| Whisper-small | ~1 GB | Loaded on demand |
+| Qwen3-TTS-1.7B (fp16) | ~3.5 GB | Loaded on demand when cloning starts |
+| Qwen2.5-3B-Instruct (4-bit) | ~2 GB | Loaded on demand for Q&A |
+| Whisper-small | ~1 GB | Loaded on demand for voice questions |
+| Supertonic (ONNX) | ~0.3 GB | Loaded on demand as fallback |
 | Gradio + PyTorch overhead | ~1–2 GB | Runtime |
-| **Total** | **~5–6 GB** | ~10 GB headroom for KV cache and activations |
+| **Total** | **~8–9 GB** | ~7 GB headroom for KV cache and activations |
 
 **Concurrency:** Single-process Gradio serializes concurrent users. The hackathon demo is single-user. For multi-user, consider Gradio queue or separate worker processes.
 
@@ -53,17 +56,19 @@ No database. No external storage. No external LLM API. Stories are flat files. A
 
 ---
 
-## 2. Voice Model (QWEN-TTS-0.6B)
+## 2. Voice Model (Qwen3-TTS-1.7B)
 
 | Aspect | Detail |
 |---|---|
-| Model | `Qwen/Qwen-TTS-0.6B` from Hugging Face Hub |
-| Size | 0.6B params — fits comfortably on T4 (16GB VRAM) |
-| Capability | Zero-shot voice cloning + TTS from text chunks |
-| Input | Reference audio (≥5s) + target text |
-| Output | WAV audio in cloned voice (24 kHz, 16-bit, mono) |
-| Latency | ~3–5s for a paragraph on T4 |
-| Optimization | Cache the voice representation after recording; generate story audio in interruptible paragraph chunks |
+| Model | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` from Hugging Face Hub |
+| Size | 1.7B params — fits on T4 in fp16 (~3.5 GB VRAM) |
+| Capability | Zero-shot voice cloning via speaker embedding extraction + TTS synthesis |
+| Input | Reference audio (≥5s) for cloning; text + cached voice profile for synthesis |
+| Output | WAV audio in cloned voice (24 kHz) |
+| Latency | ~30s for speaker embedding extraction; ~25–50s per sentence synthesis on T4 |
+| Optimization | Cache voice profile after recording (UUID-keyed server-side dict); generate story audio in sentence-level chunks streamed via background thread |
+| Fallback | Supertonic TTS with stock voices (F1–F5, M1–M5) when no voice profile exists |
+| Module | `voice_clone.py` (model + profile cache), `tts.py` (unified streaming API) |
 
 ---
 
@@ -85,7 +90,7 @@ No database. No external storage. No external LLM API. Stories are flat files. A
 | Model | `Qwen/Qwen2.5-3B-Instruct` |
 | Why | Strong small-model instruction following with lower latency and VRAM pressure than an 8B-class model |
 | Method | Current story position + relevant story passages + strict answer-from-story instruction + child question → short answer |
-| Retrieval | TF-IDF cosine similarity between the child's question and each story paragraph; return the top-2 paragraphs as context. Full-story prompt is the fallback when retrieval scores are low. |
+| Retrieval | Keyword overlap + proximity bonus: score each paragraph by shared words with the question plus a bonus for paragraphs near the current playback position; return top-3 as context. Full-story fallback when no context found. |
 | Output cap | 1–2 child-friendly sentences, typically 40–80 new tokens |
 | Runtime note | Use 4-bit/8-bit loading on T4; use bf16 or 8-bit on A10G for more headroom |
 
@@ -172,12 +177,12 @@ All other transitions are illegal. The UI should disable buttons that would trig
 ## 9. Local Dev
 
 ```bash
-pip install gradio transformers torch accelerate bitsandbytes soundfile numpy
+pip install -r requirements.txt
 python app.py
 # → http://localhost:7860
 ```
 
-No Docker, no DB, no infra setup.
+No Docker, no DB, no infra setup. Requires GPU for voice cloning and Q&A inference.
 
 ---
 
@@ -191,6 +196,10 @@ accelerate
 bitsandbytes
 soundfile
 numpy
+supertonic>=1.3.1
+onnxruntime>=1.18.0
+huggingface-hub>=0.23.0
+qwen-tts>=0.1.1
 ```
 
 ---
@@ -213,7 +222,7 @@ All in a `static/` folder loaded by gr.Server mount.
 | Area | Critique | Upgrade |
 |---|---|---|
 | Privacy | Using an external Q&A API would undermine the local-inference claim. | Qwen2.5-3B-Instruct keeps questions, story text, and generated answers inside the Space runtime. |
-| GPU fit | QWEN-TTS-0.6B, Whisper-small, and a 3B-class LLM are a more realistic fit for a single Space than an 8B-class LLM, but running all three hot can still pressure T4. | Quantize Qwen on T4, use A10G for demo headroom, cache voice/story artifacts, and load ASR only when the Ask tab receives audio. |
-| Latency | Full-story generation and spoken Q&A can feel slow if every step waits for complete outputs. | Use interruptible paragraph chunks, cache full narration, retrieve only relevant passages, and synthesize only short final answers. |
+| GPU fit | Qwen3-TTS-1.7B, Whisper-small, and Qwen2.5-3B-Instruct are a realistic fit for T4 with lazy loading and quantization, but concurrent use can pressure VRAM. | All models lazy-loaded on demand. Quantize Qwen Q&A on T4. Use A10G for demo headroom. |
+| Latency | Qwen3-TTS synthesis takes ~25–50s per sentence, making full live narration impractical. | Cache voice profile after cloning. Stream sentence chunks. Pre-generate next chunk while current plays. Keep Q&A answers to 1–2 sentences. |
 | Interaction | Streaming without cancellation can still feel rigid if the child must wait for a chunk to finish. | Add explicit playback state, Ask interruption, queued job cancellation/deprioritization, and resume from the saved chunk. |
 | Dependencies | External LLM SDKs and API secrets are no longer aligned with the model choice. | Use local inference dependencies and optional HF authentication only. |