README.md

metadata

title: AIRWAVES
emoji: 🎛️
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 6.16.0
app_file: app.py
pinned: false
license: apache-2.0
models:
  - openbmb/VoxCPM2
tags:
  - track:wood
  - sponsor:openbmb
  - achievement:offbrand
  - build-small-hackathon

🎛️ AIRWAVES

No turntables, no controller — conduct a live set with your bare hands, and a tiny AI hype-man calls your drops in real time.

Point your webcam at your hands and play the air. A raised palm screams the filter open; a tilt warps the record like vinyl; a pinch drenches it in reverb; a clenched fist builds a stutter and throwing it open slams the drop. Bend the built-in beat — or capture a browser tab (YouTube, SoundCloud) and bend any song with your hands. And riding over the top: an AI club-MC, voiced by VoxCPM2, that reacts to what you just did.

▶️ Best in Chrome on desktop. Open the direct app URL (not the embedded preview), allow the camera, and put your hands up.

---

🎮 How to play

Gesture	What it does
↕ Raise / lower a hand	Open / close the low-pass filter — pull the track up out of the floor (the hero move)
↻ Tilt your hand	Warp the record like a turntable — speed + pitch (built-in beat only)
🤏 Pinch (thumb ↔ index)	Reverb & echo — throw the sound into a cathedral
✊ Clench → throw open	Build a beat-stutter, then DROP
🙌 Two hands	Crossfade the mix (drums ↔ melody)
🎬 "Bend a tab"	Capture a Chrome tab's audio (YouTube…) and bend a real song live

---

🧠 The "her" pattern (small model used smartly)

Every real-time fact is owned by deterministic browser code — the beat clock, the exact filter Hz, the gains, the loop points, which gesture maps to what. No model is anywhere in the audio path, so the bend is instant and never glitches. The one model, VoxCPM2 (OpenBMB, 2.29B), owns only what code can't: it designs a club-MC voice from a text description (no reference audio) and speaks timed callouts that react to your performance. The hands make the music; the AI reacts to you.

🛠️ The tech we used

Vision (100% in-browser, camera never leaves your device)

• MediaPipe Tasks-Vision HandLandmarker — 21 hand landmarks/hand, two hands, GPU-with-CPU fallback, ~30fps
• Deterministic gesture math (palm height, pinch, fist-curl, wrist roll, two-hand position) → EMA-smoothed control scalars

Audio (Web Audio API, ~60fps, zero model in the realtime path)

• A reggaeton/electronic bed synthesized from scratch (OfflineAudioContext) into 3 phase-locked looping stems — no samples, no licensing
• BiquadFilterNode resonant low-pass sweep · playbackRate turntable warp · parallel ConvolverNode reverb (procedural impulse) + DelayNode echo · a look-ahead stutter gate · per-stem GainNode crossfade · AnalyserNode spectrum visualizer
• getDisplayMedia({ audio: { suppressLocalAudioPlayback } }) to capture and bend a browser tab's audio through the same effects chain

The model

• VoxCPM2 (OpenBMB) — voice-design TTS, the AI hype-man, running on ZeroGPU

Serving

• gr.Server (FastAPI) on a ZeroGPU Space. The SPA is static; the only server/GPU touch is GET /api/hype, which synthesizes the hype-man voice bank once inside a single @spaces.GPU window, caches it, and serves it to everyone instantly. Graceful fallback: if the voice is unavailable, the hype-man uses browser speech and the hand-bending is untouched.

webcam ─▶ MediaPipe hand landmarks (browser) ─▶ smoothed gesture scalars ─┐  zero latency, no model
                                                                           ▼
        3 synthesized stems / captured tab ─▶ filter ─▶ reverb · delay · stutter ─▶ master ─▶ 🔊
   ── in parallel ──
        musical-event detector (browser) ─▶ /api/hype ─▶ VoxCPM2 (ZeroGPU, cached once) ─▶ AI MC

🔒 Privacy

All computer vision and all audio run in your browser. The camera feed never leaves your device — only short text lines are sent to synthesize the hype-man's voice.

▶️ Run locally

cd web && python3 -m http.server 8777   # open http://localhost:8777 in Chrome
# Full app with the VoxCPM2 hype-man:
AIRWAVES_BACKEND=zerogpu python app.py   # (mock backend by default — a placeholder beep)

---

🏆 Build Small Hackathon

• Track: Thousand Token Wood · Sponsor model: OpenBMB VoxCPM2
• 🎬 Demo video: https://youtu.be/-jkUObdYr9U
• 🐦 Launch post: https://x.com/_Andresio/status/2066619396668449212

Model: openbmb/VoxCPM2 · Built with MediaPipe Tasks-Vision + the Web Audio API.