--- title: Whistle Coach colorFrom: green colorTo: purple sdk: gradio app_file: app.py pinned: false license: apache-2.0 tags: - build-small - backyard-ai - off-brand - tiny-titan - best-demo - openai-codex - voice - audio - audio-classification - gradio - track:backyard - track:wood - sponsor:openai - achievement:offbrand --- # Whistle Coach **Whistle Coach is an audio-first AI practice coach for one tiny human skill: learning your first whistle.** Most people learn to whistle through awkward trial and error: round the lips, blow too hard, lose the shape, try again. Whistle Coach turns that loop into a small, live, embodied AI coach. It watches only the visible lower-face posture needed for guidance, listens for breath and pitch, diagnoses the current blocker, and grows a tiny garden as the first stable whistle turns into a melody. This is intentionally small. It is not a giant general assistant. It is a narrow feedback system for one body skill, built from small models, signal processing, browser-side feature extraction, and a compact coach policy. ## Submission Links - **Demo video:** https://www.youtube.com/watch?v=GfirPNWQohc - **Social post:** [this Hugging Face Space](https://x.com/jingzi_zhao_x/status/2066270034377183446) ## Buiders - @Alphaplasti - @Zhao Jinhzi ## Why It Fits Build Small Build Small is about useful, local-feeling, tinkerable AI under the 32B parameter cap. Whistle Coach fits that spirit in four ways: 1. **A small personal problem:** learning to whistle is tiny, specific, and surprisingly hard to debug alone. 2. **Small models do real work:** the runtime audio classifier is MIT AST at 86.6M parameters; the optional coach policy path targets Nemotron Nano 4B. 3. **AI is core, not decorative:** the app converts live sensor signals into practice states, then chooses the next micro-drill. 4. **The UI is custom, playful, and not default Gradio:** the live camera stage, listening panel, coach overlays, garden reward, and melody export are a bespoke browser experience served through a Gradio Space. ## What Should Try 1. Open the Space and click **Start practice**. 2. Allow camera and microphone access. 3. Put your mouth in the dotted oval. 4. Try a soft "yuh yuh yuh" breath with rounded lips. 5. Watch the listening panel update: - airflow - whistle confidence - pitch detected - stability 6. Follow the coach's one-step correction. 7. Hold a stable pitch long enough to grow the garden and unlock a tiny melody preview. Even if the room is noisy or the microphone is imperfect, the app should still show why it is coaching the next action. ## The Core Loop ```text camera + microphone | v browser feature extraction | v practice state | v coach policy | v one micro-drill + visual cue + audio feedback | v garden progress ``` The goal is not to give a lecture about whistling. The goal is to keep the user inside a tight practice loop: observe, diagnose, adjust, try again. ## Model And Signal Stack | Layer | Role | Why it is small | | --- | --- | --- | | `MIT/ast-finetuned-audioset-10-10-0.4593` | Whistle-like audio confidence | 86.6M parameters | | `librosa.pyin` | F0 / pitch stability | deterministic signal processing | | MediaPipe Face Landmarker | visible lower-face posture guidance | browser-side visual assistant | | Nemotron Nano 4B policy path | optional coach reasoning / LoRA target | under the 32B cap | | Rule fallback | deterministic coach when no hosted policy is configured | keeps the Space reliable | The deployed app does not fake model confidence. If AST is unavailable, the UI reports that clearly and falls back to pitch/airflow coaching rather than pretending a whistle score exists. ## What Makes The AI Core Whistle Coach is not a static tutorial and not a camera filter. The app builds a structured practice state from live features: - face visible and centered - mouth opening ratio - lip roundness - mouth symmetry - jaw stability - audio airflow estimate - whistle confidence - pitch stability - stable duration The coach then maps that state to the next action: ```json { "state": "airflow_no_tone", "active_step": "narrow_air_stream", "coach_message": "You are close. Make the air stream narrower.", "overlay_cues": [{"type": "air_stream", "direction": "narrower"}], "next_drill": "narrow_air_stream" } ``` That is the AI-native part: every coaching sentence is grounded in the current attempt rather than prewritten lesson order. ## Agentic Coach Trace The optional coach policy is shaped as a compact agent: ```text Observe -> Diagnose -> Plan -> Act ``` Every response can include a trace like: ```json [ {"step": "Observe", "action": "read_state", "detail": "mouth 76%, airflow 68%, tone 18%"}, {"step": "Diagnose", "action": "airflow_no_tone", "detail": "Air is moving but pitch is not stable."}, {"step": "Plan", "action": "narrow_air_stream", "detail": "Ask for a thinner air channel."}, {"step": "Act", "action": "coach_message", "detail": "You are close. Make the air stream narrower."} ] ``` This makes the app eligible for Best Agent consideration: the coach is not just a chatbot, it is a stateful sensor-to-action loop with explicit decisions. ## Tiny Titan Angle The core runtime is intentionally tiny: - MIT AST: 86.6M parameters - optional Nemotron Nano policy: 4B target - no model in the app exceeds the 32B Build Small limit - the SFT scaffold trains a narrow LoRA adapter on structured features, not raw media The project is designed around the idea that a small, specialized model can be better for embodied coaching than a huge general model. ## Off Brand / Custom UI The experience intentionally pushes past default Gradio: - full custom HTML/CSS/JS frontend mounted inside Gradio - responsive three-panel practice layout - live camera stage with mouth target and lower-face overlays - audio listening panel with live confidence meters - garden growth reward - generated downloadable melody from the user's pitch contour The UI is meant to feel like a small practice companion, not a dashboard wrapped around a model endpoint. ## Privacy And Safety Whistle Coach is playful, but it is careful: - webcam frames stay in the browser for MediaPipe feature extraction - the app uses derived mouth-shape features for coaching, not stored face images - microphone snippets are analyzed for whistle confidence and pitch, but not saved as a dataset - the included training data stores only structured practice states and coach decisions - the app does not claim tongue detection - this is not medical, speech therapy, or professional voice training software ## Repository Map ```text app.py Gradio Space, AST loading, audio analysis, pYIN, melody API frontend/index.html custom Gradio-mounted UI shell frontend/style.css responsive custom interface frontend/whistle_coach.js camera/audio loop, garden, overlays, API calls frontend/audio_features.js browser audio feature payload helpers frontend/face_features.js mouth-shape feature payload helpers frontend/overlay_renderer.js practice-state messages and overlay cues backend/state_classifier.py deterministic practice-state classifier backend/coach_model.py optional Nemotron-compatible coach policy wrapper backend/melody_generator.py pitch contour to downloadable WAV melody data/coach_policy_sft.jsonl seed SFT data for coach policy training/finetune_nemotron_policy.py LoRA fine-tuning scaffold ``` ## Running Locally ```bash python -m venv .venv source .venv/bin/activate pip install -r requirements.txt python app.py ``` Open the local Gradio URL. Camera and microphone access require `localhost` or HTTPS in modern browsers. ## Optional Nemotron Coach Policy The Space works without external secrets. By default, the deterministic fallback keeps the live coach reliable. To enable a hosted Nemotron-compatible coach endpoint, configure these Space secrets: ```text NEMOTRON_API_URL NEMOTRON_API_KEY NEMOTRON_MODEL ``` The recommended fine-tuning target is: ```text nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 ``` The training scaffold maps: ```text practice_state history -> coach_response JSON ``` This keeps the learning problem narrow, auditable, and small. ## Known Limitations - Whistle confidence depends on microphone quality and background noise. - "Airflow" is inferred from audio energy and breath-like spectral features; it is not physical airflow measurement. - Pitch detection works best once a stable tone appears. - MediaPipe can guide visible mouth posture, but it cannot detect tongue position. - Some browsers ask for camera and microphone permissions differently; the UI is responsive, but permissions remain browser-controlled. ## Hackathon Checklist - [x] Gradio Space - [x] Custom UI beyond default Gradio - [x] Models under 32B parameters - [x] Audio-first AI loop - [x] Agent-style coach trace - [x] Privacy notes - [x] Local run instructions - [ ] Public demo video link - [ ] Public social post link - [ ] Final confirmation that requested frontmatter tags match the exact awards being entered ## Credits Built for the Build Small Hackathon. Whistle Coach was developed with OpenAI Codex as a coding partner, with the goal of showing how a small, embodied AI loop can make one tiny real-world skill easier, more delightful, and more teachable.