FIELD_NOTES.md

# You already know how to DJ: building AIRWAVES with one small voice model

*A Build Small Hackathon field report — Thousand Token Wood.*
*Live: [build-small-hackathon/airwaves](https://huggingface.co/spaces/build-small-hackathon/airwaves) · [demo video](https://youtu.be/-jkUObdYr9U)*

---

Everyone has air-DJ'd. Song comes on, hand goes up, you *feel* the filter open
on the drop even though you're just waving at the ceiling. No app lets you
actually do it.

So we built one. Point your webcam at your hands and **play the air**. Raise a
palm and the muffled track screams open from underwater to full brightness. Tilt
your hand and the record warps like vinyl. Pinch and it floods with reverb.
Clench a fist to build a stutter — then throw your hand open and the bass
**drops**. And riding over the top, a tiny AI hype-man calls it with you:

> *"¡Dale! Drop it!"*

No turntables, no MIDI controller, no mouse. Just a camera and your hands.

## The finding that shaped everything: don't generate the music

The obvious version of this app generates music with a model. We spent an hour
proving that's the wrong build. **No open music model streams in real time** —
MusicGen is autoregressive (it renders the whole clip before you hear a note),
and even the fastest diffusion models render a fixed buffer, not a *morphing*
waveform you can bend at 60fps. The "bend music with your hands" magic is not a
generation problem at all. It's a **deterministic DSP problem.**

So the realtime path has **zero model in it**. A reggaeton/electronic bed is
synthesized from scratch in the browser (`OfflineAudioContext` → three
phase-locked looping stems, no samples, no licensing), and 21 MediaPipe hand
landmarks drive a Web Audio graph directly: a resonant `BiquadFilter` sweep, a
`playbackRate` turntable warp, a procedural-impulse `ConvolverNode`, a feedback
`DelayNode`, and a look-ahead stutter gate. Hand goes up, filter opens, *now* —
no inference, no network, can't glitch on a demo laptop.

Which freed the one model we *do* use to do the thing only a model can.

## Small model used smartly: the voice, not the beat

The hype-man is **[openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2)**
(2.29B) — voice-*designed* from a text description, no reference audio:

> *"A booming, hyped club MC — breathless and electric, commanding the crowd."*

This is the pattern this community already celebrates, applied honestly:
**deterministic code owns every audio fact** — the beat clock, the exact filter
Hz, the gains, the loop points, *which* gesture maps to *what* — and the tiny
model owns the one thing code can't fake: a persona with a voice, reacting to
what you just did with your hands.

The trap was latency. VoxCPM2 lands a line in ~4–10s warm — far too slow to call
a drop *as it happens*. A hype-man that shouts "drop incoming" five seconds
after the drop isn't alive, it's broken. So we don't synthesize on the beat: the
Space generates the whole bank of MC lines **once** inside a single
`@spaces.GPU` window, caches it, and serves it to every visitor instantly. A
deterministic event-detector in the browser then fires the right pre-warmed line
on the exact downbeat, ducked over the mix. The model never touches the clock.

## Bend any song — and a limit worth stating

A built-in beat is a demo; *your* song is a hook. So AIRWAVES can capture a
browser tab (`getDisplayMedia` with `suppressLocalAudioPlayback`) and route a
real YouTube or SoundCloud track through the same hand-controlled effects chain.
Filter, reverb, echo, stutter, volume — all of it, on a song you love.

The honest limit: you can't time-stretch a *live* stream, so speed/pitch warp
stays on audio we own. We left that explicit in the UI rather than fake it.

## Things we measured and threw away

- **Genre tuning by ear, blind.** The procedural beat went through two
  rewrites chasing "more reggaeton." Lesson learned: if you can't audition the
  output in the loop, ship the groove that *works* and spend the time on feel
  you can verify — the gesture response.
- **Overloading one hand.** Early on, "pinch" and "fist" fought each other.
  Fixed by deriving each control from **independent** geometry (pinch =
  thumb↔index only; fist = middle/ring/pinky curl only) so they can't
  false-trigger.
- **Writing landmarks straight to audio params.** Instant zipper noise.
  Everything is EMA-smoothed and pushed via `setTargetAtTime` — non-negotiable,
  not polish.

## Architecture, on a napkin

```
webcam ─► MediaPipe HandLandmarker (browser) ─► smoothed gesture scalars ─┐  zero latency, no model
                                                                          ▼
   3 synthesized stems / a captured tab ─► filter ─► reverb · delay · stutter ─► 🔊
   ── in parallel ──
   musical-event detector (browser) ─► /api/hype ─► VoxCPM2 (ZeroGPU, cached once) ─► AI MC
```

Privacy is structural: **all** vision and **all** audio run in your browser. The
camera feed never leaves your device — only short text lines are sent to give
the hype-man its voice.

## The whole thing

One 2.29B model, well under the 32B ceiling, doing the one job a model is for.
The hands make the music; the AI reacts to *you*.

**[Play it →](https://huggingface.co/spaces/build-small-hackathon/airwaves)** ·
**[Watch the demo →](https://youtu.be/-jkUObdYr9U)** · built with MediaPipe +
the Web Audio API + [openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2).