| --- |
| title: Whistle Coach |
| colorFrom: green |
| colorTo: purple |
| sdk: gradio |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| tags: |
| - build-small |
| - backyard-ai |
| - off-brand |
| - tiny-titan |
| - best-demo |
| - openai-codex |
| - voice |
| - audio |
| - audio-classification |
| - gradio |
| - track:backyard |
| - track:wood |
| - sponsor:openai |
| - achievement:offbrand |
| --- |
| |
| # Whistle Coach |
|
|
| **Whistle Coach is an audio-first AI practice coach for one tiny human skill: learning your first whistle.** |
|
|
| Most people learn to whistle through awkward trial and error: round the lips, blow too hard, lose the shape, try again. Whistle Coach turns that loop into a small, live, embodied AI coach. It watches only the visible lower-face posture needed for guidance, listens for breath and pitch, diagnoses the current blocker, and grows a tiny garden as the first stable whistle turns into a melody. |
|
|
| This is intentionally small. It is not a giant general assistant. It is a narrow feedback system for one body skill, built from small models, signal processing, browser-side feature extraction, and a compact coach policy. |
|
|
| ## Submission Links |
|
|
| - **Demo video:** https://www.youtube.com/watch?v=GfirPNWQohc |
| - **Social post:** [this Hugging Face Space](https://x.com/jingzi_zhao_x/status/2066270034377183446) |
|
|
| ## Buiders |
| - @Alphaplasti |
| - @Zhao Jinhzi |
|
|
| ## Why It Fits Build Small |
|
|
| Build Small is about useful, local-feeling, tinkerable AI under the 32B parameter cap. Whistle Coach fits that spirit in four ways: |
|
|
| 1. **A small personal problem:** learning to whistle is tiny, specific, and surprisingly hard to debug alone. |
| 2. **Small models do real work:** the runtime audio classifier is MIT AST at 86.6M parameters; the optional coach policy path targets Nemotron Nano 4B. |
| 3. **AI is core, not decorative:** the app converts live sensor signals into practice states, then chooses the next micro-drill. |
| 4. **The UI is custom, playful, and not default Gradio:** the live camera stage, listening panel, coach overlays, garden reward, and melody export are a bespoke browser experience served through a Gradio Space. |
|
|
| ## What Should Try |
|
|
| 1. Open the Space and click **Start practice**. |
| 2. Allow camera and microphone access. |
| 3. Put your mouth in the dotted oval. |
| 4. Try a soft "yuh yuh yuh" breath with rounded lips. |
| 5. Watch the listening panel update: |
| - airflow |
| - whistle confidence |
| - pitch detected |
| - stability |
| 6. Follow the coach's one-step correction. |
| 7. Hold a stable pitch long enough to grow the garden and unlock a tiny melody preview. |
|
|
| Even if the room is noisy or the microphone is imperfect, the app should still show why it is coaching the next action. |
|
|
| ## The Core Loop |
|
|
| ```text |
| camera + microphone |
| | |
| v |
| browser feature extraction |
| | |
| v |
| practice state |
| | |
| v |
| coach policy |
| | |
| v |
| one micro-drill + visual cue + audio feedback |
| | |
| v |
| garden progress |
| ``` |
|
|
| The goal is not to give a lecture about whistling. The goal is to keep the user inside a tight practice loop: observe, diagnose, adjust, try again. |
|
|
| ## Model And Signal Stack |
|
|
| | Layer | Role | Why it is small | |
| | --- | --- | --- | |
| | `MIT/ast-finetuned-audioset-10-10-0.4593` | Whistle-like audio confidence | 86.6M parameters | |
| | `librosa.pyin` | F0 / pitch stability | deterministic signal processing | |
| | MediaPipe Face Landmarker | visible lower-face posture guidance | browser-side visual assistant | |
| | Nemotron Nano 4B policy path | optional coach reasoning / LoRA target | under the 32B cap | |
| | Rule fallback | deterministic coach when no hosted policy is configured | keeps the Space reliable | |
|
|
| The deployed app does not fake model confidence. If AST is unavailable, the UI reports that clearly and falls back to pitch/airflow coaching rather than pretending a whistle score exists. |
|
|
| ## What Makes The AI Core |
|
|
| Whistle Coach is not a static tutorial and not a camera filter. The app builds a structured practice state from live features: |
|
|
| - face visible and centered |
| - mouth opening ratio |
| - lip roundness |
| - mouth symmetry |
| - jaw stability |
| - audio airflow estimate |
| - whistle confidence |
| - pitch stability |
| - stable duration |
|
|
| The coach then maps that state to the next action: |
|
|
| ```json |
| { |
| "state": "airflow_no_tone", |
| "active_step": "narrow_air_stream", |
| "coach_message": "You are close. Make the air stream narrower.", |
| "overlay_cues": [{"type": "air_stream", "direction": "narrower"}], |
| "next_drill": "narrow_air_stream" |
| } |
| ``` |
|
|
| That is the AI-native part: every coaching sentence is grounded in the current attempt rather than prewritten lesson order. |
|
|
| ## Agentic Coach Trace |
|
|
| The optional coach policy is shaped as a compact agent: |
|
|
| ```text |
| Observe -> Diagnose -> Plan -> Act |
| ``` |
|
|
| Every response can include a trace like: |
|
|
| ```json |
| [ |
| {"step": "Observe", "action": "read_state", "detail": "mouth 76%, airflow 68%, tone 18%"}, |
| {"step": "Diagnose", "action": "airflow_no_tone", "detail": "Air is moving but pitch is not stable."}, |
| {"step": "Plan", "action": "narrow_air_stream", "detail": "Ask for a thinner air channel."}, |
| {"step": "Act", "action": "coach_message", "detail": "You are close. Make the air stream narrower."} |
| ] |
| ``` |
|
|
| This makes the app eligible for Best Agent consideration: the coach is not just a chatbot, it is a stateful sensor-to-action loop with explicit decisions. |
|
|
| ## Tiny Titan Angle |
|
|
| The core runtime is intentionally tiny: |
|
|
| - MIT AST: 86.6M parameters |
| - optional Nemotron Nano policy: 4B target |
| - no model in the app exceeds the 32B Build Small limit |
| - the SFT scaffold trains a narrow LoRA adapter on structured features, not raw media |
|
|
| The project is designed around the idea that a small, specialized model can be better for embodied coaching than a huge general model. |
|
|
| ## Off Brand / Custom UI |
|
|
| The experience intentionally pushes past default Gradio: |
|
|
| - full custom HTML/CSS/JS frontend mounted inside Gradio |
| - responsive three-panel practice layout |
| - live camera stage with mouth target and lower-face overlays |
| - audio listening panel with live confidence meters |
| - garden growth reward |
| - generated downloadable melody from the user's pitch contour |
|
|
| The UI is meant to feel like a small practice companion, not a dashboard wrapped around a model endpoint. |
|
|
| ## Privacy And Safety |
|
|
| Whistle Coach is playful, but it is careful: |
|
|
| - webcam frames stay in the browser for MediaPipe feature extraction |
| - the app uses derived mouth-shape features for coaching, not stored face images |
| - microphone snippets are analyzed for whistle confidence and pitch, but not saved as a dataset |
| - the included training data stores only structured practice states and coach decisions |
| - the app does not claim tongue detection |
| - this is not medical, speech therapy, or professional voice training software |
|
|
| ## Repository Map |
|
|
| ```text |
| app.py Gradio Space, AST loading, audio analysis, pYIN, melody API |
| frontend/index.html custom Gradio-mounted UI shell |
| frontend/style.css responsive custom interface |
| frontend/whistle_coach.js camera/audio loop, garden, overlays, API calls |
| frontend/audio_features.js browser audio feature payload helpers |
| frontend/face_features.js mouth-shape feature payload helpers |
| frontend/overlay_renderer.js practice-state messages and overlay cues |
| backend/state_classifier.py deterministic practice-state classifier |
| backend/coach_model.py optional Nemotron-compatible coach policy wrapper |
| backend/melody_generator.py pitch contour to downloadable WAV melody |
| data/coach_policy_sft.jsonl seed SFT data for coach policy |
| training/finetune_nemotron_policy.py LoRA fine-tuning scaffold |
| ``` |
|
|
| ## Running Locally |
|
|
| ```bash |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| Open the local Gradio URL. Camera and microphone access require `localhost` or HTTPS in modern browsers. |
|
|
| ## Optional Nemotron Coach Policy |
|
|
| The Space works without external secrets. By default, the deterministic fallback keeps the live coach reliable. |
|
|
| To enable a hosted Nemotron-compatible coach endpoint, configure these Space secrets: |
|
|
| ```text |
| NEMOTRON_API_URL |
| NEMOTRON_API_KEY |
| NEMOTRON_MODEL |
| ``` |
|
|
| The recommended fine-tuning target is: |
|
|
| ```text |
| nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 |
| ``` |
|
|
| The training scaffold maps: |
|
|
| ```text |
| practice_state history -> coach_response JSON |
| ``` |
|
|
| This keeps the learning problem narrow, auditable, and small. |
|
|
| ## Known Limitations |
|
|
| - Whistle confidence depends on microphone quality and background noise. |
| - "Airflow" is inferred from audio energy and breath-like spectral features; it is not physical airflow measurement. |
| - Pitch detection works best once a stable tone appears. |
| - MediaPipe can guide visible mouth posture, but it cannot detect tongue position. |
| - Some browsers ask for camera and microphone permissions differently; the UI is responsive, but permissions remain browser-controlled. |
|
|
| ## Hackathon Checklist |
|
|
| - [x] Gradio Space |
| - [x] Custom UI beyond default Gradio |
| - [x] Models under 32B parameters |
| - [x] Audio-first AI loop |
| - [x] Agent-style coach trace |
| - [x] Privacy notes |
| - [x] Local run instructions |
| - [ ] Public demo video link |
| - [ ] Public social post link |
| - [ ] Final confirmation that requested frontmatter tags match the exact awards being entered |
|
|
| ## Credits |
|
|
| Built for the Build Small Hackathon. |
|
|
| Whistle Coach was developed with OpenAI Codex as a coding partner, with the goal of showing how a small, embodied AI loop can make one tiny real-world skill easier, more delightful, and more teachable. |
|
|
