tech_stack.md

Tech Stack

Architecture (Hackathon-Simple)

Single Python file Gradio app. Everything runs in one process on a GPU-enabled HF Space.

┌─────────────────────────────────────────────────┐
│  Hugging Face Space (GPU: T4 or A10G)           │
│                                                 │
│  app.py (Gradio + gr.Server)                    │
│    ├── UI: Stitch-styled tabs/cards             │
│    ├── QWEN-TTS-0.6B (voice clone + chunks)     │
│    ├── Whisper-small (ASR for child questions)  │
│    └── Qwen2.5-3B-Instruct (story Q&A)          │
│                                                 │
│  stories/ (3–5 .txt files, public domain)       │
└─────────────────────────────────────────────────┘

No database. No external storage. No external LLM API. Stories are flat files. Audio is generated as interruptible chunks, then cached in the session for replay and resume.

VRAM Budget (T4 — 16 GB):

Component	Estimated VRAM	Notes
QWEN-TTS-0.6B	~1.2 GB	Always loaded
Qwen2.5-3B-Instruct (4-bit)	~2 GB	Always loaded
Whisper-small	~1 GB	Loaded on demand
Gradio + PyTorch overhead	~1–2 GB	Runtime
Total	~5–6 GB	~10 GB headroom for KV cache and activations

Concurrency: Single-process Gradio serializes concurrent users. The hackathon demo is single-user. For multi-user, consider Gradio queue or separate worker processes.

---

1. Front-End

Choice	Why
Gradio 5.x	Zero frontend code, instant HF Space deploy
gr.Server	Custom CSS/JS for Stitch-style polish (animations, palette, layout)
gr.Audio	Record/upload parent voice sample
gr.Dropdown / gr.Gallery	Story selection with cover art
gr.Audio (output)	Playback of streamed story chunks / Q&A answer
gr.Textbox + gr.Audio (input)	Child question via text or voice
Play/Pause/Ask buttons	Manual interruption and resume without open-mic barge-in

UI Tabs:

1. 🎤 Clone Voice — record/upload 15–30s, preview clone
2. 📖 Listen — pick story, hear streamed chunks in cloned voice
3. ❓ Ask — pause narration, ask about the story, hear answer, resume

---

2. Voice Model (QWEN-TTS-0.6B)

Aspect	Detail
Model	Qwen/Qwen-TTS-0.6B from Hugging Face Hub
Size	0.6B params — fits comfortably on T4 (16GB VRAM)
Capability	Zero-shot voice cloning + TTS from text chunks
Input	Reference audio (≥5s) + target text
Output	WAV audio in cloned voice (24 kHz, 16-bit, mono)
Latency	~3–5s for a paragraph on T4
Optimization	Cache the voice representation after recording; generate story audio in interruptible paragraph chunks

---

3. ASR (Child Voice Input)

Choice	Detail
Model	Whisper-small checkpoint via Transformers (local, 244M params)
Why	Fast, accurate for short child utterances; fits in GPU alongside TTS
Load strategy	Load only when the Ask tab receives audio; text questions bypass ASR
Fallback	Whisper-tiny/base or browser transcription if GPU memory or latency is tight

---

4. Q&A (Story Comprehension)

Choice	Detail
Model	Qwen/Qwen2.5-3B-Instruct
Why	Strong small-model instruction following with lower latency and VRAM pressure than an 8B-class model
Method	Current story position + relevant story passages + strict answer-from-story instruction + child question → short answer
Retrieval	TF-IDF cosine similarity between the child's question and each story paragraph; return the top-2 paragraphs as context. Full-story prompt is the fallback when retrieval scores are low.
Output cap	1–2 child-friendly sentences, typically 40–80 new tokens
Runtime note	Use 4-bit/8-bit loading on T4; use bf16 or 8-bit on A10G for more headroom

---

5. Stories (Content)

10 public domain children's stories stored as .txt in stories/, sourced from Project Gutenberg via the story_downloader/ pipeline:

Story	Words	Author/Tradition
The Tale of Peter Rabbit	948	Beatrix Potter
The Tale of Benjamin Bunny	1,118	Beatrix Potter
The Tale of Jemima Puddle-Duck	1,245	Beatrix Potter
The Tale of Tom Kitten	691	Beatrix Potter
The History of Tom Thumb	2,912	Traditional
The Story of the Three Little Pigs	956	Traditional
The Little Red Hen	1,295	Traditional
The Little Gingerbread Man	1,823	Traditional
The Sleeping Beauty	1,783	Traditional
The Adventures of Puss in Boots	503	Traditional (verse)

Each file: title on line 1, blank line, then story prose — ready for direct TTS chunking. No metadata DB needed.

Story Pipeline (story_downloader/):

• gutenberg_downloader.py — reusable downloader/parser for Project Gutenberg texts
• download_stories.py — fetches 10 specific children's stories by Gutenberg ID
• clean_stories.py — strips Gutenberg headers/footers, illustration tags, and metadata for TTS-clean output

---

6. Deployment

What	How
Platform	Hugging Face Spaces
SDK	Gradio
Hardware	T4 with quantized Qwen for the budget path; A10G for lower risk live demos
Deploy	git push to HF Space repo
Secrets	None for LLM inference; HF_TOKEN only if any selected model requires gated access
Domain	huggingface.co/spaces/{user}/readbookmom

---

7. Latency Plan

Flow	Target	Implementation
Voice setup	One-time after recording	Compute and cache the voice representation before story generation.
Story narration start	First streamed chunk in ≤ 5s	Split the story into paragraph chunks; synthesize and play the first chunk first.
Narration interruption	Pause in ≤ 500ms after Ask tap	Stop playback, preserve current chunk index, and cancel or deprioritize queued narration jobs.
Q&A interruption loop	Spoken answer starts in ≤ 8s	Use current story position, retrieve relevant passages, cap answer length, then synthesize the final answer.
Story resume	≤ 1s when next chunk is cached	Resume from the paused chunk or the next queued chunk after the answer finishes.
Story replay	Immediate after first generation	Cache generated audio by voice session and story ID.
Child audio transcription	1–2s target	Load ASR only for audio questions; prefer lighter ASR fallback for demo mode.
Q&A text answer	1–3s target	Send only relevant story passages to Qwen and cap output tokens.
Spoken Q&A answer	≤ 8s total target	Synthesize only the final short answer, not intermediate reasoning or context.

8. Interaction State

State	Meaning	Key Data
playing	Story chunk is currently playing.	story_id, voice_session_id, current_chunk_index
paused	Playback is paused by user action.	Current chunk, elapsed position if available
asking	Narration is interrupted while the child asks a question.	Current chunk, relevant passages, pending ASR input
answering	Qwen answer or answer TTS is being generated.	Question text, short answer, answer audio path
resuming	Answer finished and story playback is restarting.	Resume chunk index, cached next chunk
finished	Story narration completed.	Cached full-story audio

Legal transitions:

playing → paused → playing
playing → asking → answering → resuming → playing
playing → finished
paused  → asking  → answering → resuming → playing
asking  → asking  (child asks a follow-up before answer starts)

All other transitions are illegal. The UI should disable buttons that would trigger an illegal transition.

---

9. Local Dev

pip install gradio transformers torch accelerate bitsandbytes soundfile numpy
python app.py
# → http://localhost:7860

No Docker, no DB, no infra setup.

---

10. Dependencies

gradio>=5.0
transformers
torch
accelerate
bitsandbytes
soundfile
numpy

---

11. Google Stitch UI Customization (via gr.Server)

gr.Server injects custom HTML/CSS/JS to achieve Stitch-quality polish:

• Custom CSS: Rounded cards, warm color palette (#FFB347 accent, #FFF8E7 background), playful fonts (Nunito/Fredoka)
• Micro-animations: Fade-in on story cards, pulse on recording button, waveform visualization
• Layout overrides: Full-bleed hero on clone tab, grid gallery for stories
• Custom favicon + title: Branded for demo presentation

All in a static/ folder loaded by gr.Server mount.

---

12. Review Notes

Area	Critique	Upgrade
Privacy	Using an external Q&A API would undermine the local-inference claim.	Qwen2.5-3B-Instruct keeps questions, story text, and generated answers inside the Space runtime.
GPU fit	QWEN-TTS-0.6B, Whisper-small, and a 3B-class LLM are a more realistic fit for a single Space than an 8B-class LLM, but running all three hot can still pressure T4.	Quantize Qwen on T4, use A10G for demo headroom, cache voice/story artifacts, and load ASR only when the Ask tab receives audio.
Latency	Full-story generation and spoken Q&A can feel slow if every step waits for complete outputs.	Use interruptible paragraph chunks, cache full narration, retrieve only relevant passages, and synthesize only short final answers.
Interaction	Streaming without cancellation can still feel rigid if the child must wait for a chunk to finish.	Add explicit playback state, Ask interruption, queued job cancellation/deprioritization, and resume from the saved chunk.
Dependencies	External LLM SDKs and API secrets are no longer aligned with the model choice.	Use local inference dependencies and optional HF authentication only.