FIELD_NOTES.md

You already know how to DJ: building AIRWAVES with one small voice model

A Build Small Hackathon field report — Thousand Token Wood. Live: build-small-hackathon/airwaves · demo video

---

Everyone has air-DJ'd. Song comes on, hand goes up, you feel the filter open on the drop even though you're just waving at the ceiling. No app lets you actually do it.

So we built one. Point your webcam at your hands and play the air. Raise a palm and the muffled track screams open from underwater to full brightness. Tilt your hand and the record warps like vinyl. Pinch and it floods with reverb. Clench a fist to build a stutter — then throw your hand open and the bass drops. And riding over the top, a tiny AI hype-man calls it with you:

"¡Dale! Drop it!"

No turntables, no MIDI controller, no mouse. Just a camera and your hands.

The finding that shaped everything: don't generate the music

The obvious version of this app generates music with a model. We spent an hour proving that's the wrong build. No open music model streams in real time — MusicGen is autoregressive (it renders the whole clip before you hear a note), and even the fastest diffusion models render a fixed buffer, not a morphing waveform you can bend at 60fps. The "bend music with your hands" magic is not a generation problem at all. It's a deterministic DSP problem.

So the realtime path has zero model in it. A reggaeton/electronic bed is synthesized from scratch in the browser (OfflineAudioContext → three phase-locked looping stems, no samples, no licensing), and 21 MediaPipe hand landmarks drive a Web Audio graph directly: a resonant BiquadFilter sweep, a playbackRate turntable warp, a procedural-impulse ConvolverNode, a feedback DelayNode, and a look-ahead stutter gate. Hand goes up, filter opens, now — no inference, no network, can't glitch on a demo laptop.

Which freed the one model we do use to do the thing only a model can.

Small model used smartly: the voice, not the beat

The hype-man is openbmb/VoxCPM2 (2.29B) — voice-designed from a text description, no reference audio:

"A booming, hyped club MC — breathless and electric, commanding the crowd."

This is the pattern this community already celebrates, applied honestly: deterministic code owns every audio fact — the beat clock, the exact filter Hz, the gains, the loop points, which gesture maps to what — and the tiny model owns the one thing code can't fake: a persona with a voice, reacting to what you just did with your hands.

The trap was latency. VoxCPM2 lands a line in ~4–10s warm — far too slow to call a drop as it happens. A hype-man that shouts "drop incoming" five seconds after the drop isn't alive, it's broken. So we don't synthesize on the beat: the Space generates the whole bank of MC lines once inside a single @spaces.GPU window, caches it, and serves it to every visitor instantly. A deterministic event-detector in the browser then fires the right pre-warmed line on the exact downbeat, ducked over the mix. The model never touches the clock.

Bend any song — and a limit worth stating

A built-in beat is a demo; your song is a hook. So AIRWAVES can capture a browser tab (getDisplayMedia with suppressLocalAudioPlayback) and route a real YouTube or SoundCloud track through the same hand-controlled effects chain. Filter, reverb, echo, stutter, volume — all of it, on a song you love.

The honest limit: you can't time-stretch a live stream, so speed/pitch warp stays on audio we own. We left that explicit in the UI rather than fake it.

Things we measured and threw away

• Genre tuning by ear, blind. The procedural beat went through two rewrites chasing "more reggaeton." Lesson learned: if you can't audition the output in the loop, ship the groove that works and spend the time on feel you can verify — the gesture response.
• Overloading one hand. Early on, "pinch" and "fist" fought each other. Fixed by deriving each control from independent geometry (pinch = thumb↔index only; fist = middle/ring/pinky curl only) so they can't false-trigger.
• Writing landmarks straight to audio params. Instant zipper noise. Everything is EMA-smoothed and pushed via setTargetAtTime — non-negotiable, not polish.

Architecture, on a napkin

webcam ─► MediaPipe HandLandmarker (browser) ─► smoothed gesture scalars ─┐  zero latency, no model
                                                                          ▼
   3 synthesized stems / a captured tab ─► filter ─► reverb · delay · stutter ─► 🔊
   ── in parallel ──
   musical-event detector (browser) ─► /api/hype ─► VoxCPM2 (ZeroGPU, cached once) ─► AI MC

Privacy is structural: all vision and all audio run in your browser. The camera feed never leaves your device — only short text lines are sent to give the hype-man its voice.

The whole thing

One 2.29B model, well under the 32B ceiling, doing the one job a model is for. The hands make the music; the AI reacts to you.

Play it → · Watch the demo → · built with MediaPipe + the Web Audio API + openbmb/VoxCPM2.