sprint.md

# Sprint Plan — 2-Day Hackathon

## Current Progress (Updated 2026-06-13 10:00 PM)

**Completed: 19/25 tasks (76%)**

✅ Core ML pipeline: Voice cloning (Qwen3-TTS), ASR (Whisper-small), Q&A (Qwen2.5-3B-Instruct)
✅ Gradio UI: 4 tabs, book grid, player, clone studio, profile
✅ Latency optimizations: bfloat16, flash-attn, streaming, reduced sampling, torch.compile
✅ UI cleanup: removed dead widgets/debug tools, tightened integration
✅ Tests: 92 passing (18 latency + 74 integration)
✅ Documentation: README, SPEC, tech_stack, sprint, mission all updated

**Remaining: 6 tasks — see [Remaining Tasks](#remaining-tasks) below**

---

## Goal
Ship a public Hugging Face Space: parent clones voice → story streams in that voice → child can interrupt, ask, hear the answer, and resume.

---

## Day 1: Core Pipeline (Get It Working)

### Morning (4h) — Voice Clone + TTS

| # | Task | Time | Done |
|---|---|---|---|
| 1 | Set up repo: `app.py`, `requirements.txt`, `stories/` | 30m | ☑ |
| 2 | Load QWEN-TTS-0.6B locally, test basic TTS (text → audio) | 1h | ☑ | *(switched to Supertonic TTS; wired in `tts.py`, tested in `test_modules/`)* |
| 3 | Implement voice cloning and cache voice representation after recording | 1.5h | ☑ | *(Qwen3-TTS voice cloning in `voice_clone.py`; `create_voice_profile()` extracts speaker embedding, cached server-side; preview synthesis on clone; profile_id flows via `gr.State` to narration + Q&A)* |
| 4 | Add 10 short stories as `.txt` files (public domain, from Project Gutenberg) | 30m | ☑ |
| 5 | Wire up: pick story → stream first narration chunk, track chunk index, cache story audio | 1h | ☑ | *(`handle_book_select` loads paragraphs + splits chunks; `stream_tts` generator streams audio with play/pause/chunk-index tracking)* |

**Checkpoint:** Can generate a story audio file in a cloned voice from CLI.

### Afternoon (4h) — Gradio UI + Q&A

| # | Task | Time | Done |
|---|---|---|---|
| 6 | Build Gradio app with 3 tabs (Clone, Listen, Ask) | 1h | ☑ | *(4 tabs: Explore, Library+Player, Clone Voice Studio, Profile & Sandbox)* |
| 7 | Tab 1 (Clone): `gr.Audio` record/upload + preview button | 45m | ☑ | *(Clone Voice Studio tab: mic recorder, extract button, status pipeline, preview audio)* |
| 8 | Tab 2 (Listen): story dropdown + play/pause/resume controls for streamed chunks | 45m | ☑ | *(Library tab: book card grid + integrated player panel with play/pause/resume/chunk status)* |
| 9 | Add on-demand ASR for child voice input; use lighter ASR fallback if needed | 30m | ☑ | *(Whisper-small loaded on-demand in `inference.py`; transcribes child audio in `handle_question_submit`)* |
| 10 | Tab 3 (Ask): interrupt narration → short grounded Qwen answer → TTS → resume story | 2h | ☑ | *(Qwen2.5-3B-Instruct in `inference.py` generates grounded 1-2 sentence answers from story context; answer synthesized via Supertonic TTS; full pipeline wired in `handle_question_submit`)* |
| 10a | Pre-generate 2–3 anticipated Q&A pairs per chunk during narration playback (background task) | 30m | ☐ |

**Checkpoint:** Full loop works locally — clone → listen → interrupt → ask → resume. Ugly but functional.

**Day 1 Overflow Plan:** If the afternoon runs past 8h, cut Task 9 (on-demand ASR) and rely on text-only questions. CSS polish (Tasks 12–14) is the next cut candidate on Day 2.

---

## Day 2: Polish + Deploy (Make It Demo-Ready)

### Morning (4h) — UI Polish (Google Stitch)

| # | Task | Time | Done |
|---|---|---|---|
| 11 | Set up `gr.Server` with custom static files (`static/style.css`) | 30m | ☑ | *(`css_code` loaded from `static/style.css` in `app.py`; `allowed_paths` set for `assets/`)* |
| 12 | Stitch-style CSS: color palette, card layout, rounded corners, fonts | 1h | ☑ | *(179-line `static/style.css`: Inter/Playfair Display fonts, cream/amber palette, card grid, hero spotlight, progress bar)* |
| 13 | Recording UX: waveform animation, countdown timer, status indicators | 1h | ☐ | *(pulse-ring CSS animation + status indicators done; no countdown timer yet)* |
| 14 | Story gallery: cover images (AI-generated or placeholder), card grid | 45m | ☑ | *(10 cover PNGs in `assets/covers/`; `generate_library_html` renders card grid with cover, progress bar, narrator)* |
| 15 | Loading states: playback status, paused state, TTS progress, and background chunk generation | 30m | ☑ | *(PLAYING/PAUSED/READY/DONE status bars + chunk counter HTML; `stream_tts` generator updates per chunk)* |
| 16 | Mobile-responsive check (parents use laptops, demo on projector) | 15m | ☐ |

**Checkpoint:** App looks polished and professional. Ready for live demo.

### Afternoon (4h) — Deploy + Demo Prep

| # | Task | Time | Done |
|---|---|---|---|
| 17 | Create HF Space (T4 with 4-bit/8-bit loading, or A10G for headroom), push code, verify model downloads | 1h | ☐ |
| 18 | Confirm no external LLM API secrets are required; set `HF_TOKEN` only if gated models require it | 5m | ☑ | *(all inference local via Supertonic + Qwen; no external API calls in codebase)* |
| 19 | End-to-end test on live Space (clone → listen → interrupt → ask → resume) | 30m | ☐ |
| 20 | Fix latency issues: preload weights, cache voice/story audio, cap Q&A tokens, validate pause/resume | 45m | ☑ | *(bfloat16/flash-attn/streaming/reduced sampling params in `voice_clone.py`; torch.compile on decoder; ref audio trimming; 0.6B CustomVoice as fast fallback)* |
| 20a | Measure and log all latency targets from mission.md (first chunk, pause, Q&A, resume, replay) | 30m | ☐ |
| 20b | Validate playback state machine: test all 6 states and legal transitions from tech_stack.md | 30m | ☑ | *(92 tests in `test_modules/`: state machine consistency, handler wiring, dead code removal verified)* |
| 21 | Add error handling: graceful failures, loading messages | 30m | ☐ |
| 22 | Record backup demo video (in case live demo fails) | 30m | ☐ |
| 23 | Write README.md for the Space (screenshot, description) | 15m | ☑ | *(full README with HF Space YAML header, model table, story list, architecture, privacy section)* |
| 24 | Practice 3-minute demo walkthrough | 30m | ☐ |

**Checkpoint:** Live public URL works. Demo rehearsed.

---

## Risk Mitigations

| Risk | Mitigation |
|---|---|
| Qwen3-TTS voice quality insufficient | Fall back to longer reference audio (30s+); use Supertonic stock voice as backup |
| GPU OOM with Qwen3-TTS + Whisper + Qwen Q&A | All models lazy-loaded on demand; 4-bit Qwen Q&A on T4; prefer A10G for live demo |
| HF Space cold start too slow | Use persistent Space (not sleep); lazy-load models on first use rather than at startup |
| Qwen3-TTS synthesis too slow (~30s/sentence) | ☑ Mitigated: bfloat16, flash-attn, streaming mode, top_k=20/temp=0.7, max_new_tokens=1024, torch.compile, ref audio trimmed to ≤10s |
| Qwen answer latency too slow | Keep answers to 1–2 sentences; cap story prompt length; cache answers per story/question pair |
| Interrupt/resume feels brittle | Track playback state, current chunk index, queued generation jobs, and cached next chunks |
| ASR adds too much delay | Let text questions bypass ASR; use Whisper-tiny/base or browser transcription for demo mode |
| Live demo fails | Pre-recorded backup video ready |

---

## File Structure (Final)

```
VoiceClone/
├── app.py                  # Main Gradio app (UI + wiring)
├── voice_clone.py          # Qwen3-TTS voice cloning + profile cache
├── tts.py                  # Unified TTS interface (Qwen3 or Supertonic)
├── inference.py            # ASR (Whisper-small) + Q&A (Qwen2.5-3B-Instruct)
├── requirements.txt        # Python deps
├── README.md              # HF Space description
├── stories/                # 10 cleaned TTS-ready story texts
│   ├── The_Tale_of_Peter_Rabbit.txt
│   ├── The_Tale_of_Benjamin_Bunny.txt
│   ├── The_Tale_of_Jemima_Puddle_Duck.txt
│   ├── The_Tale_of_Tom_Kitten.txt
│   ├── The_History_of_Tom_Thumb.txt
│   ├── The_Story_of_the_Three_Little_Pigs.txt
│   ├── The_Little_Red_Hen.txt
│   ├── The_Little_Gingerbread_Man.txt
│   ├── The_Sleeping_Beauty.txt
│   └── The_Adventures_of_Puss_in_Boots.txt
├── story_downloader/       # Story acquisition & cleaning pipeline
│   ├── gutenberg_downloader.py  # Reusable Project Gutenberg downloader/parser
│   ├── download_stories.py      # Downloads 10 children's stories
│   └── clean_stories.py         # Strips Gutenberg boilerplate for TTS
├── static/
│   ├── style.css          # Stitch-style custom CSS
│   ├── script.js          # Animations, waveform
│   └── favicon.png
├── assets/
│   └── covers/            # Story cover images
├── test_modules/           # Component and integration tests
├── mission.md             # Product vision (this hackathon)
├── tech_stack.md          # Technical decisions
├── sprint.md              # This file
└── future_mobile_app_considerations.md  # Mobile deployment guidance
```

---

## Key Decisions (Locked)

- **No database** — stateless demo, voice profiles held in server-side memory cache
- **No auth** — open access for hackathon judges
- **Interruptible chunked streaming** — play sentence chunks, pause on Ask, answer, then resume from the saved story position
- **Qwen3-TTS-1.7B for voice cloning** — zero-shot speaker embedding extraction from recorded audio; synthesis in cloned voice
- **Supertonic TTS as fallback** — fast stock voice when no voice profile exists
- **Qwen2.5-3B-Instruct for Q&A** — keeps story comprehension local while improving latency and GPU fit
- **Short grounded Q&A** — retrieve relevant story passages, cap answers to 1–2 sentences, then synthesize audio
- **Button-based interruption first** — tap Ask to pause narration; always-listening voice barge-in is out of scope for the hackathon
- **Modular design** — `voice_clone.py`, `tts.py`, `inference.py` separated from `app.py` for clarity

---

## Review Notes

| Area | Critique | Upgrade |
|---|---|---|
| Model plan | Replacing an external Q&A API with Qwen2.5-3B-Instruct removes API risk while keeping latency practical. | Treat T4 with quantization as viable; use A10G when demo reliability matters more than cost. |
| Secrets | The old plan required an external LLM API secret, which contradicts the privacy-first positioning. | Remove LLM API secrets from deployment; use only HF credentials when required for model access. |
| Demo schedule | The original plan left little time for model loading, user-visible wait states, and interruption edge cases. | Add explicit validation around model downloads, quantization, cold-start behavior, chunked playback, pause/resume, and cache hits. |

---

## Remaining Tasks

| # | Task | Est. | Priority | Notes |
|---|---|---|---|---|
| 10a | Pre-generate anticipated Q&A pairs per chunk during narration | 30m | Low | Nice-to-have; reduces Q&A latency but not critical for demo |
| 13 | Recording UX: countdown timer in Clone Studio | 30m | Medium | Pulse animation done; timer would improve UX |
| 16 | Mobile-responsive check | 15m | Low | Demo is on laptop/projector; mobile is stretch goal |
| 17 | Create HF Space, push, verify model downloads | 1h | **Critical** | Must do before demo — deploy to T4/A10G Space |
| 19 | End-to-end test on live Space | 30m | **Critical** | Full clone→listen→ask→resume loop on deployed Space |
| 20a | Measure and log latency targets | 30m | Medium | Document actual vs target for first-chunk, Q&A, resume |
| 21 | Error handling: graceful failures, loading messages | 30m | Medium | Model load failures, OOM, timeout messaging |
| 22 | Record backup demo video | 30m | **Critical** | Insurance against live demo failure |
| 24 | Practice 3-minute demo walkthrough | 30m | **Critical** | Rehearse narrative + live interaction flow |

**Critical path:** 17 → 19 → 22 → 24 (deploy → test → backup video → rehearse)