tech_stack.md

# Tech Stack

## Architecture (Hackathon-Simple)

Multi-module Gradio app. Everything runs in one process on a GPU-enabled HF Space.

```
┌──────────────────────────────────────────────────────┐
│  Hugging Face Space (GPU: T4 or A10G)                │
│                                                      │
│  app.py (Gradio UI + wiring)                         │
│    ├── voice_clone.py                                │
│    │     └── Qwen3-TTS-1.7B (voice clone + TTS)      │
│    ├── tts.py (unified TTS: Qwen3 or Supertonic)     │
│    │     └── Supertonic TTS (stock voice fallback)   │
│    ├── inference.py                                  │
│    │     ├── Whisper-small (ASR for child questions)  │
│    │     └── Qwen2.5-3B-Instruct (story Q&A)         │
│    └── stories/ (10 .txt files, public domain)       │
└──────────────────────────────────────────────────────┘
```

No database. No external storage. No external LLM API. Stories are flat files. Audio is generated as interruptible chunks, then cached in the session for replay and resume.

**VRAM Budget (T4 — 16 GB):**

| Component | Estimated VRAM | Notes |
|---|---|---|
| Qwen3-TTS-1.7B (fp16) | ~3.5 GB | Loaded on demand when cloning starts |
| Qwen2.5-3B-Instruct (4-bit) | ~2 GB | Loaded on demand for Q&A |
| Whisper-small | ~1 GB | Loaded on demand for voice questions |
| Supertonic (ONNX) | ~0.3 GB | Loaded on demand as fallback |
| Gradio + PyTorch overhead | ~1–2 GB | Runtime |
| **Total** | **~8–9 GB** | ~7 GB headroom for KV cache and activations |

**Concurrency:** Single-process Gradio serializes concurrent users. The hackathon demo is single-user. For multi-user, consider Gradio queue or separate worker processes.

---

## 1. Front-End

| Choice | Why |
|---|---|
| Gradio 5.x | Zero frontend code, instant HF Space deploy |
| `gr.Server` | Custom CSS/JS for Stitch-style polish (animations, palette, layout) |
| `gr.Audio` | Record/upload parent voice sample |
| `gr.Dropdown` / `gr.Gallery` | Story selection with cover art |
| `gr.Audio` (output) | Playback of streamed story chunks / Q&A answer |
| `gr.Textbox` + `gr.Audio` (input) | Child question via text or voice |
| Play/Pause/Ask buttons | Manual interruption and resume without open-mic barge-in |

**UI Tabs:**
1. **🎤 Clone Voice** — record/upload 15–30s, preview clone
2. **📖 Listen** — pick story, hear streamed chunks in cloned voice
3. **❓ Ask** — pause narration, ask about the story, hear answer, resume

---

## 2. Voice Model (Qwen3-TTS-1.7B)

| Aspect | Detail |
|---|---|
| Model | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` from Hugging Face Hub |
| Size | 1.7B params — fits on T4 in fp16 (~3.5 GB VRAM) |
| Capability | Zero-shot voice cloning via speaker embedding extraction + TTS synthesis |
| Input | Reference audio (≥5s) for cloning; text + cached voice profile for synthesis |
| Output | WAV audio in cloned voice (24 kHz) |
| Latency | ~30s for speaker embedding extraction; ~25–50s per sentence synthesis on T4 |
| Optimization | Cache voice profile after recording (UUID-keyed server-side dict); generate story audio in sentence-level chunks streamed via background thread |
| Fallback | Supertonic TTS with stock voices (F1–F5, M1–M5) when no voice profile exists |
| Module | `voice_clone.py` (model + profile cache), `tts.py` (unified streaming API) |

---

## 3. ASR (Child Voice Input)

| Choice | Detail |
|---|---|
| Model | Whisper-small checkpoint via Transformers (local, 244M params) |
| Why | Fast, accurate for short child utterances; fits in GPU alongside TTS |
| Load strategy | Load only when the Ask tab receives audio; text questions bypass ASR |
| Fallback | Whisper-tiny/base or browser transcription if GPU memory or latency is tight |

---

## 4. Q&A (Story Comprehension)

| Choice | Detail |
|---|---|
| Model | `Qwen/Qwen2.5-3B-Instruct` |
| Why | Strong small-model instruction following with lower latency and VRAM pressure than an 8B-class model |
| Method | Current story position + relevant story passages + strict answer-from-story instruction + child question → short answer |
| Retrieval | Keyword overlap + proximity bonus: score each paragraph by shared words with the question plus a bonus for paragraphs near the current playback position; return top-3 as context. Full-story fallback when no context found. |
| Output cap | 1–2 child-friendly sentences, typically 40–80 new tokens |
| Runtime note | Use 4-bit/8-bit loading on T4; use bf16 or 8-bit on A10G for more headroom |

---

## 5. Stories (Content)

10 public domain children's stories stored as `.txt` in `stories/`, sourced from [Project Gutenberg](https://www.gutenberg.org/) via the `story_downloader/` pipeline:

| Story | Words | Author/Tradition |
|---|---|---|
| The Tale of Peter Rabbit | 948 | Beatrix Potter |
| The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
| The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
| The Tale of Tom Kitten | 691 | Beatrix Potter |
| The History of Tom Thumb | 2,912 | Traditional |
| The Story of the Three Little Pigs | 956 | Traditional |
| The Little Red Hen | 1,295 | Traditional |
| The Little Gingerbread Man | 1,823 | Traditional |
| The Sleeping Beauty | 1,783 | Traditional |
| The Adventures of Puss in Boots | 503 | Traditional (verse) |

Each file: title on line 1, blank line, then story prose — ready for direct TTS chunking. No metadata DB needed.

**Story Pipeline (`story_downloader/`):**
- `gutenberg_downloader.py` — reusable downloader/parser for Project Gutenberg texts
- `download_stories.py` — fetches 10 specific children's stories by Gutenberg ID
- `clean_stories.py` — strips Gutenberg headers/footers, illustration tags, and metadata for TTS-clean output

---

## 6. Deployment

| What | How |
|---|---|
| Platform | Hugging Face Spaces |
| SDK | Gradio |
| Hardware | T4 with quantized Qwen for the budget path; A10G for lower risk live demos |
| Deploy | `git push` to HF Space repo |
| Secrets | None for LLM inference; `HF_TOKEN` only if any selected model requires gated access |
| Domain | `huggingface.co/spaces/{user}/readbookmom` |

---

## 7. Latency Plan

| Flow | Target | Implementation |
|---|---|---|
| Voice setup | One-time after recording | Compute and cache the voice representation before story generation. |
| Story narration start | First streamed chunk in ≤ 5s | Split the story into paragraph chunks; synthesize and play the first chunk first. |
| Narration interruption | Pause in ≤ 500ms after Ask tap | Stop playback, preserve current chunk index, and cancel or deprioritize queued narration jobs. |
| Q&A interruption loop | Spoken answer starts in ≤ 8s | Use current story position, retrieve relevant passages, cap answer length, then synthesize the final answer. |
| Story resume | ≤ 1s when next chunk is cached | Resume from the paused chunk or the next queued chunk after the answer finishes. |
| Story replay | Immediate after first generation | Cache generated audio by voice session and story ID. |
| Child audio transcription | 1–2s target | Load ASR only for audio questions; prefer lighter ASR fallback for demo mode. |
| Q&A text answer | 1–3s target | Send only relevant story passages to Qwen and cap output tokens. |
| Spoken Q&A answer | ≤ 8s total target | Synthesize only the final short answer, not intermediate reasoning or context. |

## 8. Interaction State

| State | Meaning | Key Data |
|---|---|---|
| `playing` | Story chunk is currently playing. | `story_id`, `voice_session_id`, `current_chunk_index` |
| `paused` | Playback is paused by user action. | Current chunk, elapsed position if available |
| `asking` | Narration is interrupted while the child asks a question. | Current chunk, relevant passages, pending ASR input |
| `answering` | Qwen answer or answer TTS is being generated. | Question text, short answer, answer audio path |
| `resuming` | Answer finished and story playback is restarting. | Resume chunk index, cached next chunk |
| `finished` | Story narration completed. | Cached full-story audio |

**Legal transitions:**

```
playing → paused → playing
playing → asking → answering → resuming → playing
playing → finished
paused  → asking  → answering → resuming → playing
asking  → asking  (child asks a follow-up before answer starts)
```

All other transitions are illegal. The UI should disable buttons that would trigger an illegal transition.

---

## 9. Local Dev

```bash
pip install -r requirements.txt
python app.py
# → http://localhost:7860
```

No Docker, no DB, no infra setup. Requires GPU for voice cloning and Q&A inference.

---

## 10. Dependencies

```
gradio>=5.0
transformers
torch
accelerate
bitsandbytes
soundfile
numpy
supertonic>=1.3.1
onnxruntime>=1.18.0
huggingface-hub>=0.23.0
qwen-tts>=0.1.1
```

---

## 11. Google Stitch UI Customization (via gr.Server)

`gr.Server` injects custom HTML/CSS/JS to achieve Stitch-quality polish:

- **Custom CSS**: Rounded cards, warm color palette (#FFB347 accent, #FFF8E7 background), playful fonts (Nunito/Fredoka)
- **Micro-animations**: Fade-in on story cards, pulse on recording button, waveform visualization
- **Layout overrides**: Full-bleed hero on clone tab, grid gallery for stories
- **Custom favicon + title**: Branded for demo presentation

All in a `static/` folder loaded by gr.Server mount.

---

## 12. Review Notes

| Area | Critique | Upgrade |
|---|---|---|
| Privacy | Using an external Q&A API would undermine the local-inference claim. | Qwen2.5-3B-Instruct keeps questions, story text, and generated answers inside the Space runtime. |
| GPU fit | Qwen3-TTS-1.7B, Whisper-small, and Qwen2.5-3B-Instruct are a realistic fit for T4 with lazy loading and quantization, but concurrent use can pressure VRAM. | All models lazy-loaded on demand. Quantize Qwen Q&A on T4. Use A10G for demo headroom. |
| Latency | Qwen3-TTS synthesis takes ~25–50s per sentence, making full live narration impractical. | Cache voice profile after cloning. Stream sentence chunks. Pre-generate next chunk while current plays. Keep Q&A answers to 1–2 sentences. |
| Interaction | Streaming without cancellation can still feel rigid if the child must wait for a chunk to finish. | Add explicit playback state, Ask interruption, queued job cancellation/deprioritization, and resume from the saved chunk. |
| Dependencies | External LLM SDKs and API secrets are no longer aligned with the model choice. | Use local inference dependencies and optional HF authentication only. |