Spaces:
Sleeping
Sleeping
| title: AIRWAVES | |
| emoji: 🎛️ | |
| colorFrom: indigo | |
| colorTo: pink | |
| sdk: gradio | |
| sdk_version: 6.16.0 | |
| app_file: app.py | |
| pinned: false | |
| license: apache-2.0 | |
| models: | |
| - openbmb/VoxCPM2 | |
| tags: | |
| - track:wood | |
| - sponsor:openbmb | |
| - achievement:offbrand | |
| - build-small-hackathon | |
| # 🎛️ AIRWAVES | |
| **No turntables, no controller — conduct a live set with your bare hands, and a tiny AI hype-man calls your drops in real time.** | |
| Point your webcam at your hands and *play the air*. A raised palm screams the filter open; a tilt warps the record like vinyl; a pinch drenches it in reverb; a clenched fist builds a stutter and throwing it open slams the **drop**. Bend the built-in beat — or **capture a browser tab (YouTube, SoundCloud) and bend any song with your hands.** And riding over the top: an **AI club-MC, voiced by VoxCPM2**, that reacts to what you just did. | |
| > ▶️ **Best in Chrome on desktop.** Open the direct app URL (not the embedded preview), allow the camera, and put your hands up. | |
| --- | |
| ## 🎮 How to play | |
| | Gesture | What it does | | |
| |---|---| | |
| | ↕ **Raise / lower a hand** | Open / close the low-pass filter — pull the track up out of the floor *(the hero move)* | | |
| | ↻ **Tilt your hand** | Warp the record like a turntable — speed + pitch *(built-in beat only)* | | |
| | 🤏 **Pinch** (thumb ↔ index) | Reverb & echo — throw the sound into a cathedral | | |
| | ✊ **Clench → throw open** | Build a beat-stutter, then **DROP** | | |
| | 🙌 **Two hands** | Crossfade the mix (drums ↔ melody) | | |
| | 🎬 **"Bend a tab"** | Capture a Chrome tab's audio (YouTube…) and bend a real song live | | |
| --- | |
| ## 🧠 The "her" pattern (small model used smartly) | |
| Every real-time *fact* is owned by deterministic browser code — the beat clock, the exact filter Hz, the gains, the loop points, which gesture maps to what. **No model is anywhere in the audio path**, so the bend is instant and never glitches. The one model, **VoxCPM2 (OpenBMB, 2.29B)**, owns only what code can't: it *designs a club-MC voice from a text description* (no reference audio) and speaks timed callouts that react to your performance. The hands make the music; the AI reacts to *you*. | |
| ## 🛠️ The tech we used | |
| **Vision (100% in-browser, camera never leaves your device)** | |
| - **MediaPipe Tasks-Vision `HandLandmarker`** — 21 hand landmarks/hand, two hands, GPU-with-CPU fallback, ~30fps | |
| - Deterministic gesture math (palm height, pinch, fist-curl, wrist roll, two-hand position) → EMA-smoothed control scalars | |
| **Audio (Web Audio API, ~60fps, zero model in the realtime path)** | |
| - A reggaeton/electronic bed **synthesized from scratch** (`OfflineAudioContext`) into 3 phase-locked looping stems — no samples, no licensing | |
| - `BiquadFilterNode` resonant low-pass sweep · `playbackRate` turntable warp · parallel `ConvolverNode` reverb (procedural impulse) + `DelayNode` echo · a look-ahead **stutter gate** · per-stem `GainNode` crossfade · `AnalyserNode` spectrum visualizer | |
| - **`getDisplayMedia({ audio: { suppressLocalAudioPlayback } })`** to capture and bend a browser tab's audio through the same effects chain | |
| **The model** | |
| - **VoxCPM2** (OpenBMB) — voice-design TTS, the AI hype-man, running on **ZeroGPU** | |
| **Serving** | |
| - `gr.Server` (FastAPI) on a ZeroGPU Space. The SPA is static; the only server/GPU touch is `GET /api/hype`, which synthesizes the hype-man voice bank **once** inside a single `@spaces.GPU` window, caches it, and serves it to everyone instantly. Graceful fallback: if the voice is unavailable, the hype-man uses browser speech and the hand-bending is untouched. | |
| ``` | |
| webcam ─▶ MediaPipe hand landmarks (browser) ─▶ smoothed gesture scalars ─┐ zero latency, no model | |
| ▼ | |
| 3 synthesized stems / captured tab ─▶ filter ─▶ reverb · delay · stutter ─▶ master ─▶ 🔊 | |
| ── in parallel ── | |
| musical-event detector (browser) ─▶ /api/hype ─▶ VoxCPM2 (ZeroGPU, cached once) ─▶ AI MC | |
| ``` | |
| ## 🔒 Privacy | |
| All computer vision and all audio run in your browser. The camera feed never leaves your device — only short text lines are sent to synthesize the hype-man's voice. | |
| ## ▶️ Run locally | |
| ```bash | |
| cd web && python3 -m http.server 8777 # open http://localhost:8777 in Chrome | |
| # Full app with the VoxCPM2 hype-man: | |
| AIRWAVES_BACKEND=zerogpu python app.py # (mock backend by default — a placeholder beep) | |
| ``` | |
| --- | |
| ## 🏆 Build Small Hackathon | |
| - **Track:** Thousand Token Wood · **Sponsor model:** OpenBMB **VoxCPM2** | |
| - 🎬 **Demo video:** https://youtu.be/-jkUObdYr9U | |
| - 🐦 **Launch post:** https://x.com/_Andresio/status/2066619396668449212 | |
| Model: [openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) · Built with MediaPipe Tasks-Vision + the Web Audio API. | |