---
title: Whistle Coach
colorFrom: green
colorTo: purple
sdk: gradio
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - build-small
  - voice
  - audio
  - audio-classification
  - gradio
---

# Whistle Coach

**Whistle Coach** is a Build Small Hackathon **Voice / audio app**: an audio-first AI coach for a beginner's first whistle.

This is not a static tutorial and not a UI mockup. The app listens to each practice attempt, analyzes the latest audio window, and gives micro-feedback while the user practices.

## Core Experience

Click **Start Live Practice**, allow microphone input, and try a gentle whistle. The app updates the listening panel about once per audio window with:

- Airflow
- Whistle confidence from AST
- Pitch detected from F0
- Stability
- A next coaching tip
- Garden progress

If streaming is slow on CPU hardware, the same `analyze_audio()` function also runs when the user records or updates a microphone audio window.

## AI Model Stack

1. **MIT AST 86.6M audio model -> Whistle confidence**
   - Model: `MIT/ast-finetuned-audioset-10-10-0.4593`
   - Loaded globally in `app.py` with `transformers.pipeline("audio-classification", ...)`.
   - The app reads the top audio labels and uses the real Whistling label score when present.
   - If the model fails to load, the UI shows a clear error and does not fake confidence.
   - Browser audio windows are sent to the Gradio API endpoint `analyze_audio_window`, decoded as WAV, resampled to 16 kHz, and passed into this classifier.

2. **librosa.pyin -> Pitch detected / F0 / Stability**
   - Audio is converted to mono and resampled to 16 kHz.
   - `librosa.pyin` runs from about C4 to C7.
   - The app calculates voiced frames, mean pitch, pitch note, pitch standard deviation, stable duration, and pitch contour.

3. **MediaPipe -> Visual mouth guidance only**
   - Camera is a visual assistant for visible mouth posture and face framing.
   - Camera does not decide whether the user whistled.
   - MediaPipe/camera guidance cannot detect tongue position, so this app never claims tongue detection.

4. **Optional Nemotron coach policy -> Coaching wording**
   - `backend/coach_model.py` can call a hosted Nemotron-compatible chat endpoint when `NEMOTRON_API_URL` and `NEMOTRON_API_KEY` are configured as Space secrets.
   - Without those secrets, the Space uses a deterministic rule fallback, so the live practice experience still works.

## Feedback Rules

The coach uses real audio analysis results:

- Volume too low: "Blow a little more, but stay gentle."
- High noise with no stable pitch: "You are producing air noise. Make the lip opening smaller and soften the airflow."
- Medium whistle confidence or a short pitch: "You are close. Make the air stream narrower."
- Short pitch detected: "Tiny whistle found. Freeze this mouth shape."
- Stable pitch over one second: "Great! Hold this tone longer."
- Stable pitch with pitch contour movement: "Nice - you are changing notes. Try making a melody."

## Melody Preview

When the state reaches `stable_pitch` or `melody_ready`, the pitch contour is converted into a simple note sequence and rendered as a downloadable WAV melody. Before that, the melody preview remains locked.

## Run Locally

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py
```

Open the local Gradio URL. Camera and microphone access require `localhost` or HTTPS in modern browsers.

## Files

- `app.py` - Gradio Space, AST model loading, `analyze_audio()`, pYIN pitch tracking, feedback, melody generation.
- `requirements.txt` - runtime dependencies for Hugging Face Spaces.
- `README.md` - this Space documentation.

## Important Limitations

- This is a playful learning demo, not a medical, speech therapy, or professional voice-training product.
- Microphone airflow is inferred from audio energy and noise-like features; it is not physical airflow measurement.
- Pitch detection depends on microphone quality and room noise.
- Camera cannot detect tongue position.