tech_stack.md

Tech Stack

Architecture (Hackathon-Simple)

Multi-module Gradio app. Everything runs in one process on a GPU-enabled HF Space.

┌──────────────────────────────────────────────────────┐
│  Hugging Face Space (GPU: T4 or A10G)                │
│                                                      │
│  app.py (Gradio UI + wiring)                         │
│    ├── voice_clone.py                                │
│    │     └── Qwen3-TTS-1.7B (voice clone + TTS)      │
│    ├── tts.py (unified TTS: Qwen3 or Supertonic)     │
│    │     └── Supertonic TTS (stock voice fallback)   │
│    ├── inference.py                                  │
│    │     ├── Whisper-small (ASR for child questions)  │
│    │     └── Qwen2.5-3B-Instruct (story Q&A)         │
│    └── stories/ (10 .txt files, public domain)       │
└──────────────────────────────────────────────────────┘

No database. No external storage. No external LLM API. Stories are flat files. Audio is generated as interruptible chunks, then cached in the session for replay and resume.

VRAM Budget (T4 — 16 GB):

Component	Estimated VRAM	Notes
Qwen3-TTS-1.7B (fp16)	~3.5 GB	Loaded on demand when cloning starts
Qwen2.5-3B-Instruct (4-bit)	~2 GB	Loaded on demand for Q&A
Whisper-small	~1 GB	Loaded on demand for voice questions
Supertonic (ONNX)	~0.3 GB	Loaded on demand as fallback
Gradio + PyTorch overhead	~1–2 GB	Runtime
Total	~8–9 GB	~7 GB headroom for KV cache and activations

Concurrency: Single-process Gradio serializes concurrent users. The hackathon demo is single-user. For multi-user, consider Gradio queue or separate worker processes.

---

1. Front-End

Choice	Why
Gradio 5.x	Zero frontend code, instant HF Space deploy
gr.Server	Custom CSS/JS for Stitch-style polish (animations, palette, layout)
gr.Audio	Record/upload parent voice sample
gr.Dropdown / gr.Gallery	Story selection with cover art
gr.Audio (output)	Playback of streamed story chunks / Q&A answer
gr.Textbox + gr.Audio (input)	Child question via text or voice
Play/Pause/Ask buttons	Manual interruption and resume without open-mic barge-in

UI Tabs:

1. 🎤 Clone Voice — record/upload 15–30s, preview clone
2. 📖 Listen — pick story, hear streamed chunks in cloned voice
3. ❓ Ask — pause narration, ask about the story, hear answer, resume

---

2. Voice Model (Qwen3-TTS-1.7B)

Aspect	Detail
Model	Qwen/Qwen3-TTS-12Hz-1.7B-Base from Hugging Face Hub
Size	1.7B params — fits on T4 in fp16 (~3.5 GB VRAM)
Capability	Zero-shot voice cloning via speaker embedding extraction + TTS synthesis
Input	Reference audio (≥5s) for cloning; text + cached voice profile for synthesis
Output	WAV audio in cloned voice (24 kHz)
Latency	~30s for speaker embedding extraction; ~25–50s per sentence synthesis on T4
Optimization	Cache voice profile after recording (UUID-keyed server-side dict); generate story audio in sentence-level chunks streamed via background thread
Fallback	Supertonic TTS with stock voices (F1–F5, M1–M5) when no voice profile exists
Module	voice_clone.py (model + profile cache), tts.py (unified streaming API)

---

3. ASR (Child Voice Input)

Choice	Detail
Model	Whisper-small checkpoint via Transformers (local, 244M params)
Why	Fast, accurate for short child utterances; fits in GPU alongside TTS
Load strategy	Load only when the Ask tab receives audio; text questions bypass ASR
Fallback	Whisper-tiny/base or browser transcription if GPU memory or latency is tight

---

4. Q&A (Story Comprehension)

Choice	Detail
Model	Qwen/Qwen2.5-3B-Instruct
Why	Strong small-model instruction following with lower latency and VRAM pressure than an 8B-class model
Method	Current story position + relevant story passages + strict answer-from-story instruction + child question → short answer
Retrieval	Keyword overlap + proximity bonus: score each paragraph by shared words with the question plus a bonus for paragraphs near the current playback position; return top-3 as context. Full-story fallback when no context found.
Output cap	1–2 child-friendly sentences, typically 40–80 new tokens
Runtime note	Use 4-bit/8-bit loading on T4; use bf16 or 8-bit on A10G for more headroom

---

5. Stories (Content)

10 public domain children's stories stored as .txt in stories/, sourced from Project Gutenberg via the story_downloader/ pipeline:

Story	Words	Author/Tradition
The Tale of Peter Rabbit	948	Beatrix Potter
The Tale of Benjamin Bunny	1,118	Beatrix Potter
The Tale of Jemima Puddle-Duck	1,245	Beatrix Potter
The Tale of Tom Kitten	691	Beatrix Potter
The History of Tom Thumb	2,912	Traditional
The Story of the Three Little Pigs	956	Traditional
The Little Red Hen	1,295	Traditional
The Little Gingerbread Man	1,823	Traditional
The Sleeping Beauty	1,783	Traditional
The Adventures of Puss in Boots	503	Traditional (verse)

Each file: title on line 1, blank line, then story prose — ready for direct TTS chunking. No metadata DB needed.

Story Pipeline (story_downloader/):

• gutenberg_downloader.py — reusable downloader/parser for Project Gutenberg texts
• download_stories.py — fetches 10 specific children's stories by Gutenberg ID
• clean_stories.py — strips Gutenberg headers/footers, illustration tags, and metadata for TTS-clean output

---

6. Deployment

What	How
Platform	Hugging Face Spaces
SDK	Gradio
Hardware	T4 with quantized Qwen for the budget path; A10G for lower risk live demos
Deploy	git push to HF Space repo
Secrets	None for LLM inference; HF_TOKEN only if any selected model requires gated access
Domain	huggingface.co/spaces/{user}/readbookmom

---

7. Latency Plan

Flow	Target	Implementation
Voice setup	One-time after recording	Compute and cache the voice representation before story generation.
Story narration start	First streamed chunk in ≤ 5s	Split the story into paragraph chunks; synthesize and play the first chunk first.
Narration interruption	Pause in ≤ 500ms after Ask tap	Stop playback, preserve current chunk index, and cancel or deprioritize queued narration jobs.
Q&A interruption loop	Spoken answer starts in ≤ 8s	Use current story position, retrieve relevant passages, cap answer length, then synthesize the final answer.
Story resume	≤ 1s when next chunk is cached	Resume from the paused chunk or the next queued chunk after the answer finishes.
Story replay	Immediate after first generation	Cache generated audio by voice session and story ID.
Child audio transcription	1–2s target	Load ASR only for audio questions; prefer lighter ASR fallback for demo mode.
Q&A text answer	1–3s target	Send only relevant story passages to Qwen and cap output tokens.
Spoken Q&A answer	≤ 8s total target	Synthesize only the final short answer, not intermediate reasoning or context.

8. Interaction State

State	Meaning	Key Data
playing	Story chunk is currently playing.	story_id, voice_session_id, current_chunk_index
paused	Playback is paused by user action.	Current chunk, elapsed position if available
asking	Narration is interrupted while the child asks a question.	Current chunk, relevant passages, pending ASR input
answering	Qwen answer or answer TTS is being generated.	Question text, short answer, answer audio path
resuming	Answer finished and story playback is restarting.	Resume chunk index, cached next chunk
finished	Story narration completed.	Cached full-story audio

Legal transitions:

playing → paused → playing
playing → asking → answering → resuming → playing
playing → finished
paused  → asking  → answering → resuming → playing
asking  → asking  (child asks a follow-up before answer starts)

All other transitions are illegal. The UI should disable buttons that would trigger an illegal transition.

---

9. Local Dev

pip install -r requirements.txt
python app.py
# → http://localhost:7860

No Docker, no DB, no infra setup. Requires GPU for voice cloning and Q&A inference.

---

10. Dependencies

gradio>=5.0
transformers
torch
accelerate
bitsandbytes
soundfile
numpy
supertonic>=1.3.1
onnxruntime>=1.18.0
huggingface-hub>=0.23.0
qwen-tts>=0.1.1

---

11. Google Stitch UI Customization (via gr.Server)

gr.Server injects custom HTML/CSS/JS to achieve Stitch-quality polish:

• Custom CSS: Rounded cards, warm color palette (#FFB347 accent, #FFF8E7 background), playful fonts (Nunito/Fredoka)
• Micro-animations: Fade-in on story cards, pulse on recording button, waveform visualization
• Layout overrides: Full-bleed hero on clone tab, grid gallery for stories
• Custom favicon + title: Branded for demo presentation

All in a static/ folder loaded by gr.Server mount.

---

12. Review Notes

Area	Critique	Upgrade
Privacy	Using an external Q&A API would undermine the local-inference claim.	Qwen2.5-3B-Instruct keeps questions, story text, and generated answers inside the Space runtime.
GPU fit	Qwen3-TTS-1.7B, Whisper-small, and Qwen2.5-3B-Instruct are a realistic fit for T4 with lazy loading and quantization, but concurrent use can pressure VRAM.	All models lazy-loaded on demand. Quantize Qwen Q&A on T4. Use A10G for demo headroom.
Latency	Qwen3-TTS synthesis takes ~25–50s per sentence, making full live narration impractical.	Cache voice profile after cloning. Stream sentence chunks. Pre-generate next chunk while current plays. Keep Q&A answers to 1–2 sentences.
Interaction	Streaming without cancellation can still feel rigid if the child must wait for a chunk to finish.	Add explicit playback state, Ask interruption, queued job cancellation/deprioritization, and resume from the saved chunk.
Dependencies	External LLM SDKs and API secrets are no longer aligned with the model choice.	Use local inference dependencies and optional HF authentication only.