sprint.md

Sprint Plan — 2-Day Hackathon

Current Progress (Updated 2026-06-13 10:00 PM)

Completed: 19/25 tasks (76%)

✅ Core ML pipeline: Voice cloning (Qwen3-TTS), ASR (Whisper-small), Q&A (Qwen2.5-3B-Instruct) ✅ Gradio UI: 4 tabs, book grid, player, clone studio, profile ✅ Latency optimizations: bfloat16, flash-attn, streaming, reduced sampling, torch.compile ✅ UI cleanup: removed dead widgets/debug tools, tightened integration ✅ Tests: 92 passing (18 latency + 74 integration) ✅ Documentation: README, SPEC, tech_stack, sprint, mission all updated

Remaining: 6 tasks — see Remaining Tasks below

---

Goal

Ship a public Hugging Face Space: parent clones voice → story streams in that voice → child can interrupt, ask, hear the answer, and resume.

---

Day 1: Core Pipeline (Get It Working)

Morning (4h) — Voice Clone + TTS

#	Task	Time	Done
1	Set up repo: app.py, requirements.txt, stories/	30m	☑
2	Load QWEN-TTS-0.6B locally, test basic TTS (text → audio)	1h	☑
3	Implement voice cloning and cache voice representation after recording	1.5h	☑
4	Add 10 short stories as .txt files (public domain, from Project Gutenberg)	30m	☑
5	Wire up: pick story → stream first narration chunk, track chunk index, cache story audio	1h	☑

Checkpoint: Can generate a story audio file in a cloned voice from CLI.

Afternoon (4h) — Gradio UI + Q&A

#	Task	Time	Done
6	Build Gradio app with 3 tabs (Clone, Listen, Ask)	1h	☑
7	Tab 1 (Clone): gr.Audio record/upload + preview button	45m	☑
8	Tab 2 (Listen): story dropdown + play/pause/resume controls for streamed chunks	45m	☑
9	Add on-demand ASR for child voice input; use lighter ASR fallback if needed	30m	☑
10	Tab 3 (Ask): interrupt narration → short grounded Qwen answer → TTS → resume story	2h	☑
10a	Pre-generate 2–3 anticipated Q&A pairs per chunk during narration playback (background task)	30m	☐

Checkpoint: Full loop works locally — clone → listen → interrupt → ask → resume. Ugly but functional.

Day 1 Overflow Plan: If the afternoon runs past 8h, cut Task 9 (on-demand ASR) and rely on text-only questions. CSS polish (Tasks 12–14) is the next cut candidate on Day 2.

---

Day 2: Polish + Deploy (Make It Demo-Ready)

Morning (4h) — UI Polish (Google Stitch)

#	Task	Time	Done
11	Set up gr.Server with custom static files (static/style.css)	30m	☑
12	Stitch-style CSS: color palette, card layout, rounded corners, fonts	1h	☑
13	Recording UX: waveform animation, countdown timer, status indicators	1h	☐
14	Story gallery: cover images (AI-generated or placeholder), card grid	45m	☑
15	Loading states: playback status, paused state, TTS progress, and background chunk generation	30m	☑
16	Mobile-responsive check (parents use laptops, demo on projector)	15m	☐

Checkpoint: App looks polished and professional. Ready for live demo.

Afternoon (4h) — Deploy + Demo Prep

#	Task	Time	Done
17	Create HF Space (T4 with 4-bit/8-bit loading, or A10G for headroom), push code, verify model downloads	1h	☐
18	Confirm no external LLM API secrets are required; set HF_TOKEN only if gated models require it	5m	☑
19	End-to-end test on live Space (clone → listen → interrupt → ask → resume)	30m	☐
20	Fix latency issues: preload weights, cache voice/story audio, cap Q&A tokens, validate pause/resume	45m	☑
20a	Measure and log all latency targets from mission.md (first chunk, pause, Q&A, resume, replay)	30m	☐
20b	Validate playback state machine: test all 6 states and legal transitions from tech_stack.md	30m	☑
21	Add error handling: graceful failures, loading messages	30m	☐
22	Record backup demo video (in case live demo fails)	30m	☐
23	Write README.md for the Space (screenshot, description)	15m	☑
24	Practice 3-minute demo walkthrough	30m	☐

Checkpoint: Live public URL works. Demo rehearsed.

---

Risk Mitigations

Risk	Mitigation
Qwen3-TTS voice quality insufficient	Fall back to longer reference audio (30s+); use Supertonic stock voice as backup
GPU OOM with Qwen3-TTS + Whisper + Qwen Q&A	All models lazy-loaded on demand; 4-bit Qwen Q&A on T4; prefer A10G for live demo
HF Space cold start too slow	Use persistent Space (not sleep); lazy-load models on first use rather than at startup
Qwen3-TTS synthesis too slow (~30s/sentence)	☑ Mitigated: bfloat16, flash-attn, streaming mode, top_k=20/temp=0.7, max_new_tokens=1024, torch.compile, ref audio trimmed to ≤10s
Qwen answer latency too slow	Keep answers to 1–2 sentences; cap story prompt length; cache answers per story/question pair
Interrupt/resume feels brittle	Track playback state, current chunk index, queued generation jobs, and cached next chunks
ASR adds too much delay	Let text questions bypass ASR; use Whisper-tiny/base or browser transcription for demo mode
Live demo fails	Pre-recorded backup video ready

---

File Structure (Final)

VoiceClone/
├── app.py                  # Main Gradio app (UI + wiring)
├── voice_clone.py          # Qwen3-TTS voice cloning + profile cache
├── tts.py                  # Unified TTS interface (Qwen3 or Supertonic)
├── inference.py            # ASR (Whisper-small) + Q&A (Qwen2.5-3B-Instruct)
├── requirements.txt        # Python deps
├── README.md              # HF Space description
├── stories/                # 10 cleaned TTS-ready story texts
│   ├── The_Tale_of_Peter_Rabbit.txt
│   ├── The_Tale_of_Benjamin_Bunny.txt
│   ├── The_Tale_of_Jemima_Puddle_Duck.txt
│   ├── The_Tale_of_Tom_Kitten.txt
│   ├── The_History_of_Tom_Thumb.txt
│   ├── The_Story_of_the_Three_Little_Pigs.txt
│   ├── The_Little_Red_Hen.txt
│   ├── The_Little_Gingerbread_Man.txt
│   ├── The_Sleeping_Beauty.txt
│   └── The_Adventures_of_Puss_in_Boots.txt
├── story_downloader/       # Story acquisition & cleaning pipeline
│   ├── gutenberg_downloader.py  # Reusable Project Gutenberg downloader/parser
│   ├── download_stories.py      # Downloads 10 children's stories
│   └── clean_stories.py         # Strips Gutenberg boilerplate for TTS
├── static/
│   ├── style.css          # Stitch-style custom CSS
│   ├── script.js          # Animations, waveform
│   └── favicon.png
├── assets/
│   └── covers/            # Story cover images
├── test_modules/           # Component and integration tests
├── mission.md             # Product vision (this hackathon)
├── tech_stack.md          # Technical decisions
├── sprint.md              # This file
└── future_mobile_app_considerations.md  # Mobile deployment guidance

---

Key Decisions (Locked)

• No database — stateless demo, voice profiles held in server-side memory cache
• No auth — open access for hackathon judges
• Interruptible chunked streaming — play sentence chunks, pause on Ask, answer, then resume from the saved story position
• Qwen3-TTS-1.7B for voice cloning — zero-shot speaker embedding extraction from recorded audio; synthesis in cloned voice
• Supertonic TTS as fallback — fast stock voice when no voice profile exists
• Qwen2.5-3B-Instruct for Q&A — keeps story comprehension local while improving latency and GPU fit
• Short grounded Q&A — retrieve relevant story passages, cap answers to 1–2 sentences, then synthesize audio
• Button-based interruption first — tap Ask to pause narration; always-listening voice barge-in is out of scope for the hackathon
• Modular design — voice_clone.py, tts.py, inference.py separated from app.py for clarity

---

Review Notes

Area	Critique	Upgrade
Model plan	Replacing an external Q&A API with Qwen2.5-3B-Instruct removes API risk while keeping latency practical.	Treat T4 with quantization as viable; use A10G when demo reliability matters more than cost.
Secrets	The old plan required an external LLM API secret, which contradicts the privacy-first positioning.	Remove LLM API secrets from deployment; use only HF credentials when required for model access.
Demo schedule	The original plan left little time for model loading, user-visible wait states, and interruption edge cases.	Add explicit validation around model downloads, quantization, cold-start behavior, chunked playback, pause/resume, and cache hits.

---

Remaining Tasks

#	Task	Est.	Priority	Notes
10a	Pre-generate anticipated Q&A pairs per chunk during narration	30m	Low	Nice-to-have; reduces Q&A latency but not critical for demo
13	Recording UX: countdown timer in Clone Studio	30m	Medium	Pulse animation done; timer would improve UX
16	Mobile-responsive check	15m	Low	Demo is on laptop/projector; mobile is stretch goal
17	Create HF Space, push, verify model downloads	1h	Critical	Must do before demo — deploy to T4/A10G Space
19	End-to-end test on live Space	30m	Critical	Full clone→listen→ask→resume loop on deployed Space
20a	Measure and log latency targets	30m	Medium	Document actual vs target for first-chunk, Q&A, resume
21	Error handling: graceful failures, loading messages	30m	Medium	Model load failures, OOM, timeout messaging
22	Record backup demo video	30m	Critical	Insurance against live demo failure
24	Practice 3-minute demo walkthrough	30m	Critical	Rehearse narrative + live interaction flow

Critical path: 17 → 19 → 22 → 24 (deploy → test → backup video → rehearse)