--- title: AIRWAVES emoji: ๐ŸŽ›๏ธ colorFrom: indigo colorTo: pink sdk: gradio sdk_version: 6.16.0 app_file: app.py pinned: false license: apache-2.0 models: - openbmb/VoxCPM2 tags: - track:wood - sponsor:openbmb - achievement:offbrand - build-small-hackathon --- # ๐ŸŽ›๏ธ AIRWAVES **No turntables, no controller โ€” conduct a live set with your bare hands, and a tiny AI hype-man calls your drops in real time.** Point your webcam at your hands and *play the air*. A raised palm screams the filter open; a tilt warps the record like vinyl; a pinch drenches it in reverb; a clenched fist builds a stutter and throwing it open slams the **drop**. Bend the built-in beat โ€” or **capture a browser tab (YouTube, SoundCloud) and bend any song with your hands.** And riding over the top: an **AI club-MC, voiced by VoxCPM2**, that reacts to what you just did. > โ–ถ๏ธ **Best in Chrome on desktop.** Open the direct app URL (not the embedded preview), allow the camera, and put your hands up. --- ## ๐ŸŽฎ How to play | Gesture | What it does | |---|---| | โ†• **Raise / lower a hand** | Open / close the low-pass filter โ€” pull the track up out of the floor *(the hero move)* | | โ†ป **Tilt your hand** | Warp the record like a turntable โ€” speed + pitch *(built-in beat only)* | | ๐Ÿค **Pinch** (thumb โ†” index) | Reverb & echo โ€” throw the sound into a cathedral | | โœŠ **Clench โ†’ throw open** | Build a beat-stutter, then **DROP** | | ๐Ÿ™Œ **Two hands** | Crossfade the mix (drums โ†” melody) | | ๐ŸŽฌ **"Bend a tab"** | Capture a Chrome tab's audio (YouTubeโ€ฆ) and bend a real song live | --- ## ๐Ÿง  The "her" pattern (small model used smartly) Every real-time *fact* is owned by deterministic browser code โ€” the beat clock, the exact filter Hz, the gains, the loop points, which gesture maps to what. **No model is anywhere in the audio path**, so the bend is instant and never glitches. The one model, **VoxCPM2 (OpenBMB, 2.29B)**, owns only what code can't: it *designs a club-MC voice from a text description* (no reference audio) and speaks timed callouts that react to your performance. The hands make the music; the AI reacts to *you*. ## ๐Ÿ› ๏ธ The tech we used **Vision (100% in-browser, camera never leaves your device)** - **MediaPipe Tasks-Vision `HandLandmarker`** โ€” 21 hand landmarks/hand, two hands, GPU-with-CPU fallback, ~30fps - Deterministic gesture math (palm height, pinch, fist-curl, wrist roll, two-hand position) โ†’ EMA-smoothed control scalars **Audio (Web Audio API, ~60fps, zero model in the realtime path)** - A reggaeton/electronic bed **synthesized from scratch** (`OfflineAudioContext`) into 3 phase-locked looping stems โ€” no samples, no licensing - `BiquadFilterNode` resonant low-pass sweep ยท `playbackRate` turntable warp ยท parallel `ConvolverNode` reverb (procedural impulse) + `DelayNode` echo ยท a look-ahead **stutter gate** ยท per-stem `GainNode` crossfade ยท `AnalyserNode` spectrum visualizer - **`getDisplayMedia({ audio: { suppressLocalAudioPlayback } })`** to capture and bend a browser tab's audio through the same effects chain **The model** - **VoxCPM2** (OpenBMB) โ€” voice-design TTS, the AI hype-man, running on **ZeroGPU** **Serving** - `gr.Server` (FastAPI) on a ZeroGPU Space. The SPA is static; the only server/GPU touch is `GET /api/hype`, which synthesizes the hype-man voice bank **once** inside a single `@spaces.GPU` window, caches it, and serves it to everyone instantly. Graceful fallback: if the voice is unavailable, the hype-man uses browser speech and the hand-bending is untouched. ``` webcam โ”€โ–ถ MediaPipe hand landmarks (browser) โ”€โ–ถ smoothed gesture scalars โ”€โ” zero latency, no model โ–ผ 3 synthesized stems / captured tab โ”€โ–ถ filter โ”€โ–ถ reverb ยท delay ยท stutter โ”€โ–ถ master โ”€โ–ถ ๐Ÿ”Š โ”€โ”€ in parallel โ”€โ”€ musical-event detector (browser) โ”€โ–ถ /api/hype โ”€โ–ถ VoxCPM2 (ZeroGPU, cached once) โ”€โ–ถ AI MC ``` ## ๐Ÿ”’ Privacy All computer vision and all audio run in your browser. The camera feed never leaves your device โ€” only short text lines are sent to synthesize the hype-man's voice. ## โ–ถ๏ธ Run locally ```bash cd web && python3 -m http.server 8777 # open http://localhost:8777 in Chrome # Full app with the VoxCPM2 hype-man: AIRWAVES_BACKEND=zerogpu python app.py # (mock backend by default โ€” a placeholder beep) ``` --- ## ๐Ÿ† Build Small Hackathon - **Track:** Thousand Token Wood ยท **Sponsor model:** OpenBMB **VoxCPM2** - ๐ŸŽฌ **Demo video:** https://youtu.be/-jkUObdYr9U - ๐Ÿฆ **Launch post:** https://x.com/_Andresio/status/2066619396668449212 Model: [openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) ยท Built with MediaPipe Tasks-Vision + the Web Audio API.