| ---
|
| title: Whistle Coach
|
| colorFrom: green
|
| colorTo: purple
|
| sdk: gradio
|
| app_file: app.py
|
| pinned: false
|
| license: apache-2.0
|
| tags:
|
| - build-small
|
| - voice
|
| - audio
|
| - audio-classification
|
| - gradio
|
| ---
|
|
|
| # Whistle Coach
|
|
|
| **Whistle Coach** is a Build Small Hackathon **Voice / audio app**: an audio-first AI coach for a beginner's first whistle.
|
|
|
| This is not a static tutorial and not a UI mockup. The app listens to each practice attempt, analyzes the latest audio window, and gives micro-feedback while the user practices.
|
|
|
| ## Core Experience
|
|
|
| Click **Start Live Practice**, allow microphone input, and try a gentle whistle. The app updates the listening panel about once per audio window with:
|
|
|
| - Airflow
|
| - Whistle confidence from AST
|
| - Pitch detected from F0
|
| - Stability
|
| - A next coaching tip
|
| - Garden progress
|
|
|
| If streaming is slow on CPU hardware, the same `analyze_audio()` function also runs when the user records or updates a microphone audio window.
|
|
|
| ## AI Model Stack
|
|
|
| 1. **MIT AST 86.6M audio model -> Whistle confidence** |
| - Model: `MIT/ast-finetuned-audioset-10-10-0.4593` |
| - Loaded globally in `app.py` with `transformers.pipeline("audio-classification", ...)`. |
| - The app reads the top audio labels and uses the real Whistling label score when present. |
| - If the model fails to load, the UI shows a clear error and does not fake confidence. |
| - Browser audio windows are sent to the Gradio API endpoint `analyze_audio_window`, decoded as WAV, resampled to 16 kHz, and passed into this classifier. |
|
|
| 2. **librosa.pyin -> Pitch detected / F0 / Stability**
|
| - Audio is converted to mono and resampled to 16 kHz.
|
| - `librosa.pyin` runs from about C4 to C7.
|
| - The app calculates voiced frames, mean pitch, pitch note, pitch standard deviation, stable duration, and pitch contour.
|
|
|
| 3. **MediaPipe -> Visual mouth guidance only** |
| - Camera is a visual assistant for visible mouth posture and face framing. |
| - Camera does not decide whether the user whistled. |
| - MediaPipe/camera guidance cannot detect tongue position, so this app never claims tongue detection. |
|
|
| 4. **Optional Nemotron coach policy -> Coaching wording** |
| - `backend/coach_model.py` can call a hosted Nemotron-compatible chat endpoint when `NEMOTRON_API_URL` and `NEMOTRON_API_KEY` are configured as Space secrets. |
| - Without those secrets, the Space uses a deterministic rule fallback, so the live practice experience still works. |
|
|
| ## Feedback Rules
|
|
|
| The coach uses real audio analysis results:
|
|
|
| - Volume too low: "Blow a little more, but stay gentle." |
| - High noise with no stable pitch: "You are producing air noise. Make the lip opening smaller and soften the airflow." |
| - Medium whistle confidence or a short pitch: "You are close. Make the air stream narrower." |
| - Short pitch detected: "Tiny whistle found. Freeze this mouth shape." |
| - Stable pitch over one second: "Great! Hold this tone longer." |
| - Stable pitch with pitch contour movement: "Nice - you are changing notes. Try making a melody." |
|
|
| ## Melody Preview
|
|
|
| When the state reaches `stable_pitch` or `melody_ready`, the pitch contour is converted into a simple note sequence and rendered as a downloadable WAV melody. Before that, the melody preview remains locked.
|
|
|
| ## Run Locally
|
|
|
| ```bash
|
| python -m venv .venv
|
| source .venv/bin/activate
|
| pip install -r requirements.txt
|
| python app.py
|
| ```
|
|
|
| Open the local Gradio URL. Camera and microphone access require `localhost` or HTTPS in modern browsers.
|
|
|
| ## Files
|
|
|
| - `app.py` - Gradio Space, AST model loading, `analyze_audio()`, pYIN pitch tracking, feedback, melody generation.
|
| - `requirements.txt` - runtime dependencies for Hugging Face Spaces.
|
| - `README.md` - this Space documentation.
|
|
|
| ## Important Limitations
|
|
|
| - This is a playful learning demo, not a medical, speech therapy, or professional voice-training product.
|
| - Microphone airflow is inferred from audio energy and noise-like features; it is not physical airflow measurement.
|
| - Pitch detection depends on microphone quality and room noise.
|
| - Camera cannot detect tongue position.
|
|
|
