README.md

metadata

title: Whistle Coach
colorFrom: green
colorTo: purple
sdk: gradio
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - build-small
  - voice
  - audio
  - audio-classification
  - gradio

Whistle Coach

Whistle Coach is a Build Small Hackathon Voice / audio app: an audio-first AI coach for a beginner's first whistle.

This is not a static tutorial and not a UI mockup. The app listens to each practice attempt, analyzes the latest audio window, and gives micro-feedback while the user practices.

Core Experience

Click Start Live Practice, allow microphone input, and try a gentle whistle. The app updates the listening panel about once per audio window with:

• Airflow
• Whistle confidence from AST
• Pitch detected from F0
• Stability
• A next coaching tip
• Garden progress

If streaming is slow on CPU hardware, the same analyze_audio() function also runs when the user records or updates a microphone audio window.

AI Model Stack

1. MIT AST 86.6M audio model -> Whistle confidence

   • Model: MIT/ast-finetuned-audioset-10-10-0.4593
   • Loaded globally in app.py with transformers.pipeline("audio-classification", ...).
   • The app reads the top audio labels and uses the real Whistling label score when present.
   • If the model fails to load, the UI shows a clear error and does not fake confidence.
   • Browser audio windows are sent to the Gradio API endpoint analyze_audio_window, decoded as WAV, resampled to 16 kHz, and passed into this classifier.

2. librosa.pyin -> Pitch detected / F0 / Stability

   • Audio is converted to mono and resampled to 16 kHz.
   • librosa.pyin runs from about C4 to C7.
   • The app calculates voiced frames, mean pitch, pitch note, pitch standard deviation, stable duration, and pitch contour.

3. MediaPipe -> Visual mouth guidance only

   • Camera is a visual assistant for visible mouth posture and face framing.
   • Camera does not decide whether the user whistled.
   • MediaPipe/camera guidance cannot detect tongue position, so this app never claims tongue detection.

4. Optional Nemotron coach policy -> Coaching wording

   • backend/coach_model.py can call a hosted Nemotron-compatible chat endpoint when NEMOTRON_API_URL and NEMOTRON_API_KEY are configured as Space secrets.
   • Without those secrets, the Space uses a deterministic rule fallback, so the live practice experience still works.

Feedback Rules

The coach uses real audio analysis results:

• Volume too low: "Blow a little more, but stay gentle."
• High noise with no stable pitch: "You are producing air noise. Make the lip opening smaller and soften the airflow."
• Medium whistle confidence or a short pitch: "You are close. Make the air stream narrower."
• Short pitch detected: "Tiny whistle found. Freeze this mouth shape."
• Stable pitch over one second: "Great! Hold this tone longer."
• Stable pitch with pitch contour movement: "Nice - you are changing notes. Try making a melody."

Melody Preview

When the state reaches stable_pitch or melody_ready, the pitch contour is converted into a simple note sequence and rendered as a downloadable WAV melody. Before that, the melody preview remains locked.

Run Locally

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

Open the local Gradio URL. Camera and microphone access require localhost or HTTPS in modern browsers.

Files

• app.py - Gradio Space, AST model loading, analyze_audio(), pYIN pitch tracking, feedback, melody generation.
• requirements.txt - runtime dependencies for Hugging Face Spaces.
• README.md - this Space documentation.

Important Limitations

• This is a playful learning demo, not a medical, speech therapy, or professional voice-training product.
• Microphone airflow is inferred from audio energy and noise-like features; it is not physical airflow measurement.
• Pitch detection depends on microphone quality and room noise.
• Camera cannot detect tongue position.