README.md

metadata

title: Whistle Coach
colorFrom: green
colorTo: purple
sdk: gradio
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - build-small
  - backyard-ai
  - off-brand
  - tiny-titan
  - best-demo
  - openai-codex
  - voice
  - audio
  - audio-classification
  - gradio
  - track:backyard
  - track:wood
  - sponsor:openai
  - achievement:offbrand

Whistle Coach

Whistle Coach is an audio-first AI practice coach for one tiny human skill: learning your first whistle.

Most people learn to whistle through awkward trial and error: round the lips, blow too hard, lose the shape, try again. Whistle Coach turns that loop into a small, live, embodied AI coach. It watches only the visible lower-face posture needed for guidance, listens for breath and pitch, diagnoses the current blocker, and grows a tiny garden as the first stable whistle turns into a melody.

This is intentionally small. It is not a giant general assistant. It is a narrow feedback system for one body skill, built from small models, signal processing, browser-side feature extraction, and a compact coach policy.

Submission Links

• Demo video: https://www.youtube.com/watch?v=GfirPNWQohc
• Social post: this Hugging Face Space

Buiders

• @Alphaplasti
• @Zhao Jinhzi

Why It Fits Build Small

Build Small is about useful, local-feeling, tinkerable AI under the 32B parameter cap. Whistle Coach fits that spirit in four ways:

1. A small personal problem: learning to whistle is tiny, specific, and surprisingly hard to debug alone.
2. Small models do real work: the runtime audio classifier is MIT AST at 86.6M parameters; the optional coach policy path targets Nemotron Nano 4B.
3. AI is core, not decorative: the app converts live sensor signals into practice states, then chooses the next micro-drill.
4. The UI is custom, playful, and not default Gradio: the live camera stage, listening panel, coach overlays, garden reward, and melody export are a bespoke browser experience served through a Gradio Space.

What Should Try

1. Open the Space and click Start practice.
2. Allow camera and microphone access.
3. Put your mouth in the dotted oval.
4. Try a soft "yuh yuh yuh" breath with rounded lips.
5. Watch the listening panel update:
   • airflow
   • whistle confidence
   • pitch detected
   • stability
6. Follow the coach's one-step correction.
7. Hold a stable pitch long enough to grow the garden and unlock a tiny melody preview.

Even if the room is noisy or the microphone is imperfect, the app should still show why it is coaching the next action.

The Core Loop

camera + microphone
        |
        v
browser feature extraction
        |
        v
practice state
        |
        v
coach policy
        |
        v
one micro-drill + visual cue + audio feedback
        |
        v
garden progress

The goal is not to give a lecture about whistling. The goal is to keep the user inside a tight practice loop: observe, diagnose, adjust, try again.

Model And Signal Stack

Layer	Role	Why it is small
MIT/ast-finetuned-audioset-10-10-0.4593	Whistle-like audio confidence	86.6M parameters
librosa.pyin	F0 / pitch stability	deterministic signal processing
MediaPipe Face Landmarker	visible lower-face posture guidance	browser-side visual assistant
Nemotron Nano 4B policy path	optional coach reasoning / LoRA target	under the 32B cap
Rule fallback	deterministic coach when no hosted policy is configured	keeps the Space reliable

The deployed app does not fake model confidence. If AST is unavailable, the UI reports that clearly and falls back to pitch/airflow coaching rather than pretending a whistle score exists.

What Makes The AI Core

Whistle Coach is not a static tutorial and not a camera filter. The app builds a structured practice state from live features:

• face visible and centered
• mouth opening ratio
• lip roundness
• mouth symmetry
• jaw stability
• audio airflow estimate
• whistle confidence
• pitch stability
• stable duration

The coach then maps that state to the next action:

{
  "state": "airflow_no_tone",
  "active_step": "narrow_air_stream",
  "coach_message": "You are close. Make the air stream narrower.",
  "overlay_cues": [{"type": "air_stream", "direction": "narrower"}],
  "next_drill": "narrow_air_stream"
}

That is the AI-native part: every coaching sentence is grounded in the current attempt rather than prewritten lesson order.

Agentic Coach Trace

The optional coach policy is shaped as a compact agent:

Observe -> Diagnose -> Plan -> Act

Every response can include a trace like:

[
  {"step": "Observe", "action": "read_state", "detail": "mouth 76%, airflow 68%, tone 18%"},
  {"step": "Diagnose", "action": "airflow_no_tone", "detail": "Air is moving but pitch is not stable."},
  {"step": "Plan", "action": "narrow_air_stream", "detail": "Ask for a thinner air channel."},
  {"step": "Act", "action": "coach_message", "detail": "You are close. Make the air stream narrower."}
]

This makes the app eligible for Best Agent consideration: the coach is not just a chatbot, it is a stateful sensor-to-action loop with explicit decisions.

Tiny Titan Angle

The core runtime is intentionally tiny:

• MIT AST: 86.6M parameters
• optional Nemotron Nano policy: 4B target
• no model in the app exceeds the 32B Build Small limit
• the SFT scaffold trains a narrow LoRA adapter on structured features, not raw media

The project is designed around the idea that a small, specialized model can be better for embodied coaching than a huge general model.

Off Brand / Custom UI

The experience intentionally pushes past default Gradio:

• full custom HTML/CSS/JS frontend mounted inside Gradio
• responsive three-panel practice layout
• live camera stage with mouth target and lower-face overlays
• audio listening panel with live confidence meters
• garden growth reward
• generated downloadable melody from the user's pitch contour

The UI is meant to feel like a small practice companion, not a dashboard wrapped around a model endpoint.

Privacy And Safety

Whistle Coach is playful, but it is careful:

• webcam frames stay in the browser for MediaPipe feature extraction
• the app uses derived mouth-shape features for coaching, not stored face images
• microphone snippets are analyzed for whistle confidence and pitch, but not saved as a dataset
• the included training data stores only structured practice states and coach decisions
• the app does not claim tongue detection
• this is not medical, speech therapy, or professional voice training software

Repository Map

app.py                         Gradio Space, AST loading, audio analysis, pYIN, melody API
frontend/index.html            custom Gradio-mounted UI shell
frontend/style.css             responsive custom interface
frontend/whistle_coach.js      camera/audio loop, garden, overlays, API calls
frontend/audio_features.js     browser audio feature payload helpers
frontend/face_features.js      mouth-shape feature payload helpers
frontend/overlay_renderer.js   practice-state messages and overlay cues
backend/state_classifier.py    deterministic practice-state classifier
backend/coach_model.py         optional Nemotron-compatible coach policy wrapper
backend/melody_generator.py    pitch contour to downloadable WAV melody
data/coach_policy_sft.jsonl    seed SFT data for coach policy
training/finetune_nemotron_policy.py  LoRA fine-tuning scaffold

Running Locally

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

Open the local Gradio URL. Camera and microphone access require localhost or HTTPS in modern browsers.

Optional Nemotron Coach Policy

The Space works without external secrets. By default, the deterministic fallback keeps the live coach reliable.

To enable a hosted Nemotron-compatible coach endpoint, configure these Space secrets:

NEMOTRON_API_URL
NEMOTRON_API_KEY
NEMOTRON_MODEL

The recommended fine-tuning target is:

nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

The training scaffold maps:

practice_state history -> coach_response JSON

This keeps the learning problem narrow, auditable, and small.

Known Limitations

• Whistle confidence depends on microphone quality and background noise.
• "Airflow" is inferred from audio energy and breath-like spectral features; it is not physical airflow measurement.
• Pitch detection works best once a stable tone appears.
• MediaPipe can guide visible mouth posture, but it cannot detect tongue position.
• Some browsers ask for camera and microphone permissions differently; the UI is responsive, but permissions remain browser-controlled.

Hackathon Checklist

• Gradio Space
• Custom UI beyond default Gradio
• Models under 32B parameters
• Audio-first AI loop
• Agent-style coach trace
• Privacy notes
• Local run instructions
• Public demo video link
• Public social post link
• Final confirmation that requested frontmatter tags match the exact awards being entered

Credits

Built for the Build Small Hackathon.

Whistle Coach was developed with OpenAI Codex as a coding partner, with the goal of showing how a small, embodied AI loop can make one tiny real-world skill easier, more delightful, and more teachable.