sshtel commited on
Commit
be0895c
Β·
1 Parent(s): 94d3077

fix README demo img

Browse files
Files changed (1) hide show
  1. README.md +47 -60
README.md CHANGED
@@ -43,25 +43,21 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
43
 
44
  ## How It Works
45
 
46
-
47
- | Step | What Happens | Model |
48
- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
49
- | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached. | Qwen3-TTS-1.7B Base |
50
- | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to CustomVoice 0.6B stock voice (speaker "vivian") if no clone. | Qwen3-TTS-1.7B / CustomVoice 0.6B |
51
- | **Ask** | Child taps Ask β†’ narration pauses, microphone auto-starts. When recording stops, the question is sent directly to the Q&A audio model. | LFM2.5-Audio-1.5B |
52
- | **Answer** | A short grounded answer is generated from the story context and spoken by the low-latency Q&A audio model. It does not use the cloned narrator voice. | LFM2.5-Audio-1.5B |
53
- | **Resume** | Story continues from where it left off. Play also pauses on tab switch. | Cached chunks |
54
-
55
 
56
  ## Models
57
 
58
-
59
- | Model | Role | Size |
60
- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ----------- |
61
- | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
62
- | [Qwen3-TTS-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) | Fast stock-voice TTS (predefined speakers) | 0.6B params |
63
- | [LiquidAI/LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) | End-to-end story Q&A with audio input and spoken audio output | 1.5B params |
64
-
65
 
66
  The stock narration model and LFM Q&A model are preloaded at startup for fast response times. The 1.7B cloning model is loaded when cloning or cloned narration is needed.
67
 
@@ -76,32 +72,27 @@ The stock narration model and LFM Q&A model are preloaded at startup for fast re
76
 
77
  ## System Requirements
78
 
79
-
80
- | GPU | VRAM | Status | Notes |
81
- | ------------------- | ----- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- |
82
- | Nvidia A10G (24 GB) | 24 GB | βœ… Recommended | All three models load comfortably; narration + Q&A run without memory pressure |
83
- | Nvidia T4 (16 GB) | 16 GB | ⚠️ Functional | Works but is tight β€” voice cloning model may OOM if loaded concurrently with LFM; restart may be needed between heavy operations |
84
- | CPU only | β€” | ❌ Not viable | LFM2.5-Audio and Qwen3-TTS inference is too slow for real-time use |
85
-
86
 
87
  **Hugging Face Spaces:**
88
-
89
  - **A10G Small** β€” confirmed working, recommended for the hackathon demo
90
  - **T4 Small** β€” runs with limitations; recommend disabling concurrent model loading
91
 
92
- HuggingFace Space Hardware Options
93
 
94
  **Memory breakdown (approximate):**
95
 
96
-
97
- | Model | VRAM (bfloat16) |
98
- | ----------------------------------- | --------------- |
99
- | LFM2.5-Audio-1.5B | ~3 GB |
100
- | Qwen3-TTS-0.6B-CustomVoice | ~1.2 GB |
101
- | Qwen3-TTS-1.7B-Base (cloning) | ~3.4 GB |
102
- | KV cache + activations at inference | ~1–2 GB |
103
- | **Total (all models loaded)** | **~8–9 GB** |
104
-
105
 
106
  The 1.7B cloning model is loaded on demand (not at startup), so idle VRAM usage is ~5–6 GB with only the narration and Q&A models preloaded.
107
 
@@ -152,20 +143,18 @@ python app.py
152
 
153
  ## Stories Included
154
 
155
-
156
- | Story | Words | Source |
157
- | ---------------------------------- | ----- | ------------------- |
158
- | The Tale of Peter Rabbit | 948 | Beatrix Potter |
159
- | The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
160
- | The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
161
- | The Tale of Tom Kitten | 691 | Beatrix Potter |
162
- | The History of Tom Thumb | 2,912 | Traditional |
163
- | The Story of the Three Little Pigs | 956 | Traditional |
164
- | The Little Red Hen | 1,295 | Traditional |
165
- | The Little Gingerbread Man | 1,823 | Traditional |
166
- | The Sleeping Beauty | 1,783 | Traditional |
167
- | The Adventures of Puss in Boots | 503 | Traditional (verse) |
168
-
169
 
170
  All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/).
171
 
@@ -188,6 +177,7 @@ All inference runs on the Hugging Face Space GPU. Voice samples, story text, and
188
  Hackathon project. Stories are public domain.
189
  test push verification
190
 
 
191
  ## Application Demo
192
 
193
  # MomsVoice β€” Demo Walkthrough
@@ -198,7 +188,7 @@ Step-by-step guide through the app with screenshots.
198
 
199
  ## Step 1: Explore the Story Shelf
200
 
201
- Explore Space
202
 
203
  Open the **Explore Space** tab to browse available stories displayed as warm bookshelf cards. Each card shows the cover art, title, author, and a short excerpt. Click **"Tap to Play"** on any card to open it in My Library, or click **"More Info"** for the full story description.
204
 
@@ -208,7 +198,7 @@ The featured story highlights the current pick. All 10 stories are public domain
208
 
209
  ## Step 2: Open My Library and Start Narration
210
 
211
- My Library β€” Now Playing
212
 
213
  Switch to the **My Library** tab. Click any story card to load it into the **Now Playing** panel on the right. The panel shows:
214
 
@@ -223,7 +213,7 @@ Narration starts immediately, playing one paragraph at a time using the TTS mode
223
 
224
  ## Step 3: Ask a Question About the Story
225
 
226
- Voice Q&A Panel
227
 
228
  Scroll down in the Library panel to find the **Ask About the Story** section. Click **❓ Ask a Question** β€” narration pauses automatically and the microphone starts recording.
229
 
@@ -233,12 +223,11 @@ Speak your question naturally (e.g. *"What did Puss in Boots do to help his mast
233
 
234
  ## Step 4: Hear the Spoken Answer (LFM2.5-Audio)
235
 
236
- LFM Answer Audio
237
 
238
  The recorded question waveform appears in the audio widget. Below it, the **LFM2.5-Audio** model generates a grounded answer in 1–2 sentences, based only on the current story context.
239
 
240
  The answer is delivered in two ways simultaneously:
241
-
242
  - **Answer text** shown inline (e.g. *"Puss in Boots, a poor man, bargained for boots and a bag, and Grimalkin, ailient rabbit, rescued him from a king's wrath."*)
243
  - **Answer Audio** waveform auto-plays the spoken response
244
 
@@ -248,10 +237,9 @@ After the answer finishes, narration resumes from where it paused.
248
 
249
  ## Step 5: Open Clone Voice Studio and Record Your Voice
250
 
251
- Clone Studio β€” Recording
252
 
253
  Go to the **Clone Voice Studio** tab. Fill in:
254
-
255
  - **Voice Nickname** β€” a name for this profile (e.g. *"Mom"*, *"Grandpa Joe"*)
256
  - **Voice Gender Avatar** β€” Female or Male
257
 
@@ -261,7 +249,7 @@ A story script is displayed to read aloud. Click the **Record** button and read
261
 
262
  ## Step 6: Analyze & Synthesize the Vocal Profile
263
 
264
- Clone Studio β€” Processing
265
 
266
  Click **Analyze & Synthesize Customized Vocal Profile**. The **Status Pipeline Visualizer** on the right tracks progress through extraction, de-noising, and pitch tuning stages.
267
 
@@ -271,10 +259,9 @@ A progress bar shows synthesis percentage (e.g. *"Generating voice preview... 70
271
 
272
  ## Step 7: Voice Profile Ready
273
 
274
- Clone Studio β€” Complete
275
 
276
  When synthesis completes, the Status Pipeline Visualizer shows:
277
-
278
  - Your **avatar** and nickname with a **"Cloned successfully"** badge
279
  - *"Synthesized today using Qwen3-TTS voice cloning"*
280
  - A **preview audio player** β€” tap Play to hear a short sample in your cloned voice
@@ -286,8 +273,8 @@ The profile is saved to disk and will persist across Space restarts.
286
 
287
  ## Step 8: Story Narration in Your Cloned Voice
288
 
289
- Narration with Cloned Voice
290
 
291
  Return to **My Library** and select a story. The Now Playing panel shows **Active Narrator: [your nickname]** β€” confirming the cloned voice is active.
292
 
293
- Narration plays in your recorded voice, paragraph by paragraph (*"Paragraph 1 / 32"*). The story text scrolls in the reader below the player. All TTS inference runs locally on the Space GPU β€” no audio leaves the server.
 
43
 
44
  ## How It Works
45
 
46
+ | Step | What Happens | Model |
47
+ |---|---|---|
48
+ | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached. | Qwen3-TTS-1.7B Base |
49
+ | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to CustomVoice 0.6B stock voice (speaker "vivian") if no clone. | Qwen3-TTS-1.7B / CustomVoice 0.6B |
50
+ | **Ask** | Child taps Ask β†’ narration pauses, microphone auto-starts. When recording stops, the question is sent directly to the Q&A audio model. | LFM2.5-Audio-1.5B |
51
+ | **Answer** | A short grounded answer is generated from the story context and spoken by the low-latency Q&A audio model. It does not use the cloned narrator voice. | LFM2.5-Audio-1.5B |
52
+ | **Resume** | Story continues from where it left off. Play also pauses on tab switch. | Cached chunks |
 
 
53
 
54
  ## Models
55
 
56
+ | Model | Role | Size |
57
+ |---|---|---|
58
+ | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
59
+ | [Qwen3-TTS-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) | Fast stock-voice TTS (predefined speakers) | 0.6B params |
60
+ | [LiquidAI/LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) | End-to-end story Q&A with audio input and spoken audio output | 1.5B params |
 
 
61
 
62
  The stock narration model and LFM Q&A model are preloaded at startup for fast response times. The 1.7B cloning model is loaded when cloning or cloned narration is needed.
63
 
 
72
 
73
  ## System Requirements
74
 
75
+ | GPU | VRAM | Status | Notes |
76
+ |---|---|---|---|
77
+ | Nvidia A10G (24 GB) | 24 GB | βœ… Recommended | All three models load comfortably; narration + Q&A run without memory pressure |
78
+ | Nvidia T4 (16 GB) | 16 GB | ⚠️ Functional | Works but is tight β€” voice cloning model may OOM if loaded concurrently with LFM; restart may be needed between heavy operations |
79
+ | CPU only | β€” | ❌ Not viable | LFM2.5-Audio and Qwen3-TTS inference is too slow for real-time use |
 
 
80
 
81
  **Hugging Face Spaces:**
 
82
  - **A10G Small** β€” confirmed working, recommended for the hackathon demo
83
  - **T4 Small** β€” runs with limitations; recommend disabling concurrent model loading
84
 
85
+ ![HuggingFace Space Hardware Options](docs/hf_spec.png)
86
 
87
  **Memory breakdown (approximate):**
88
 
89
+ | Model | VRAM (bfloat16) |
90
+ |---|---|
91
+ | LFM2.5-Audio-1.5B | ~3 GB |
92
+ | Qwen3-TTS-0.6B-CustomVoice | ~1.2 GB |
93
+ | Qwen3-TTS-1.7B-Base (cloning) | ~3.4 GB |
94
+ | KV cache + activations at inference | ~1–2 GB |
95
+ | **Total (all models loaded)** | **~8–9 GB** |
 
 
96
 
97
  The 1.7B cloning model is loaded on demand (not at startup), so idle VRAM usage is ~5–6 GB with only the narration and Q&A models preloaded.
98
 
 
143
 
144
  ## Stories Included
145
 
146
+ | Story | Words | Source |
147
+ |---|---|---|
148
+ | The Tale of Peter Rabbit | 948 | Beatrix Potter |
149
+ | The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
150
+ | The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
151
+ | The Tale of Tom Kitten | 691 | Beatrix Potter |
152
+ | The History of Tom Thumb | 2,912 | Traditional |
153
+ | The Story of the Three Little Pigs | 956 | Traditional |
154
+ | The Little Red Hen | 1,295 | Traditional |
155
+ | The Little Gingerbread Man | 1,823 | Traditional |
156
+ | The Sleeping Beauty | 1,783 | Traditional |
157
+ | The Adventures of Puss in Boots | 503 | Traditional (verse) |
 
 
158
 
159
  All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/).
160
 
 
177
  Hackathon project. Stories are public domain.
178
  test push verification
179
 
180
+
181
  ## Application Demo
182
 
183
  # MomsVoice β€” Demo Walkthrough
 
188
 
189
  ## Step 1: Explore the Story Shelf
190
 
191
+ ![Explore Space](docs/demo01.png)
192
 
193
  Open the **Explore Space** tab to browse available stories displayed as warm bookshelf cards. Each card shows the cover art, title, author, and a short excerpt. Click **"Tap to Play"** on any card to open it in My Library, or click **"More Info"** for the full story description.
194
 
 
198
 
199
  ## Step 2: Open My Library and Start Narration
200
 
201
+ ![My Library β€” Now Playing](docs/demo02.png)
202
 
203
  Switch to the **My Library** tab. Click any story card to load it into the **Now Playing** panel on the right. The panel shows:
204
 
 
213
 
214
  ## Step 3: Ask a Question About the Story
215
 
216
+ ![Voice Q&A Panel](docs/demo03.png)
217
 
218
  Scroll down in the Library panel to find the **Ask About the Story** section. Click **❓ Ask a Question** β€” narration pauses automatically and the microphone starts recording.
219
 
 
223
 
224
  ## Step 4: Hear the Spoken Answer (LFM2.5-Audio)
225
 
226
+ ![LFM Answer Audio](docs/demo04.png)
227
 
228
  The recorded question waveform appears in the audio widget. Below it, the **LFM2.5-Audio** model generates a grounded answer in 1–2 sentences, based only on the current story context.
229
 
230
  The answer is delivered in two ways simultaneously:
 
231
  - **Answer text** shown inline (e.g. *"Puss in Boots, a poor man, bargained for boots and a bag, and Grimalkin, ailient rabbit, rescued him from a king's wrath."*)
232
  - **Answer Audio** waveform auto-plays the spoken response
233
 
 
237
 
238
  ## Step 5: Open Clone Voice Studio and Record Your Voice
239
 
240
+ ![Clone Studio β€” Recording](docs/demo05_clone.png)
241
 
242
  Go to the **Clone Voice Studio** tab. Fill in:
 
243
  - **Voice Nickname** β€” a name for this profile (e.g. *"Mom"*, *"Grandpa Joe"*)
244
  - **Voice Gender Avatar** β€” Female or Male
245
 
 
249
 
250
  ## Step 6: Analyze & Synthesize the Vocal Profile
251
 
252
+ ![Clone Studio β€” Processing](docs/demo06_clone.png)
253
 
254
  Click **Analyze & Synthesize Customized Vocal Profile**. The **Status Pipeline Visualizer** on the right tracks progress through extraction, de-noising, and pitch tuning stages.
255
 
 
259
 
260
  ## Step 7: Voice Profile Ready
261
 
262
+ ![Clone Studio β€” Complete](docs/demo07_clone.png)
263
 
264
  When synthesis completes, the Status Pipeline Visualizer shows:
 
265
  - Your **avatar** and nickname with a **"Cloned successfully"** badge
266
  - *"Synthesized today using Qwen3-TTS voice cloning"*
267
  - A **preview audio player** β€” tap Play to hear a short sample in your cloned voice
 
273
 
274
  ## Step 8: Story Narration in Your Cloned Voice
275
 
276
+ ![Narration with Cloned Voice](docs/demo08_clone.png)
277
 
278
  Return to **My Library** and select a story. The Now Playing panel shows **Active Narrator: [your nickname]** β€” confirming the cloned voice is active.
279
 
280
+ Narration plays in your recorded voice, paragraph by paragraph (*"Paragraph 1 / 32"*). The story text scrolls in the reader below the player. All TTS inference runs locally on the Space GPU β€” no audio leaves the server.