| --- |
| title: Marquee |
| emoji: 🎙️ |
| colorFrom: purple |
| colorTo: pink |
| sdk: gradio |
| sdk_version: 6.18.0 |
| app_file: app.py |
| license: apache-2.0 |
| short_description: Turn any clip into a broadcast starring your friends |
| tags: |
| - build-small-hackathon |
| - track:backyard |
| - achievement:offgrid |
| - achievement:offbrand |
| --- |
| |
| # Marquee 🎙️ |
|
|
| Drop a clip of friends doing anything. Marquee detects who's in it, you give each person a name, pick a commentary style, and an AI commentator calls the action play-by-play — by name, in sync — as a broadcast you can play back and export as an MP4. |
|
|
| ## What it does |
|
|
| 1. **Scan** — Upload a clip (≤60s). OpenVINO detects and clusters people into tracks. You get a roster of face crops to name. |
| 2. **Generate** — Qwen2.5-VL-7B watches key moments from the clip and writes commentary lines timed to the action, in whichever style you picked. |
| 3. **Voice** — Each line gets spoken by a TTS model. Two options: Chatterbox (fast, runs on CPU) or Orpheus (slower, more expressive, requires GPU and a HuggingFace token). |
| 4. **Export** — ffmpeg mixes the TTS audio over the original clip audio, burns in subtitles, and gives you a download. |
|
|
| ## Setup |
|
|
| ### Hardware |
|
|
| Must be **ZeroGPU**. Qwen2.5-VL-7B and Orpheus both need a GPU. Set it under **Settings → Hardware → ZeroGPU** in your Space. On a CPU-only Space the VLM is unusable. |
|
|
| ### Using Orpheus TTS |
|
|
| Orpheus uses `canopylabs/orpheus-3b-0.1-ft`, which is a **gated model** on HuggingFace. Two steps to unlock it: |
|
|
| 1. Go to https://huggingface.co/canopylabs/orpheus-3b-0.1-ft and accept the access request. Takes about a minute to be approved. |
|
|
| 2. Add your HuggingFace token as a **Space secret** (not a repo file): |
| - Go to **Settings → Variables and Secrets → New Secret** |
| - Name: `HF_TOKEN` |
| - Value: a token from https://huggingface.co/settings/tokens with at least **Read** scope |
|
|
| Once the secret is set, restart the Space. The model will download on first use (~7GB). After that it's cached for the session. |
|
|
| Chatterbox (the default) needs no token and no extra setup. |
|
|
| ### Demo |
| https://youtu.be/f2I8m6ardkU |
|
|
| ### Local dev |
|
|
| ```bash |
| pip install -r requirements.txt |
| python -c "from ov_models import download_models; download_models()" |
| python app.py |
| # → http://localhost:7860 |
| ``` |
|
|
| Qwen lazy-loads on the first generate request. Orpheus lazy-loads on first TTS request with that model selected. Expect a slow first run. |
|
|
| For local Orpheus use you'll need to be logged in: `huggingface-cli login`. |
|
|
| ## How it's wired |
|
|
| The backend is a `gradio.Server` (a FastAPI subclass). This gives us ZeroGPU + the Gradio queue while letting custom routes take priority. The UI is a static HTML page served at `/`. The scan is a plain `fetch()` POST; commentary goes through the Gradio JS client so it hits the queue and ZeroGPU sees the `@spaces.GPU` decorator. |
|
|
| ``` |
| Marquee.html + marquee.css + marquee.js → served at "/" |
| │ |
| ├─ fetch POST /api/scan upload → normalize → detect+cluster → session + roster |
| ├─ fetch GET /video/{sid} streams the normalized mp4 |
| ├─ gradio client /generate key events → annotated frames → Qwen (ZeroGPU) → script |
| └─ fetch POST /api/tts script lines → Chatterbox or Orpheus (ZeroGPU) → WAV |
| └─ fetch POST /api/export WAVs + video → ffmpeg mix → MP4 download |
| ``` |
|
|
| Session state (video path, tracks, motion data) lives in a server-side dict keyed by `session_id`. Nothing heavy goes back and forth to the client between steps. |
|
|
| ## Files |
|
|
| | File | What it does | |
| |------|-------------| |
| | `app.py` | FastAPI routes, session management, UI assembly | |
| | `Marquee.html` / `marquee.css` / `marquee.js` | The UI | |
| | `commentary.py` | Qwen2.5-VL personas, prompt, JSON parse | |
| | `tts.py` | Chatterbox + Orpheus generation, collision fix, WAV encoding | |
| | `faces.py` | OpenVINO detect → cluster, frame annotation | |
| | `events.py` | Motion-based key event selection | |
| | `video.py` | Rotation-aware ffmpeg normalize to 720p | |
| | `ov_models.py` | OpenVINO IR download + runtime wrappers | |
|
|
| ## Models |
|
|
| - **Qwen2.5-VL-7B-Instruct** — commentary generation (ZeroGPU) |
| - **Chatterbox 0.5B** — default TTS, MIT license, CPU inference |
| - **Orpheus 3B** (`canopylabs/orpheus-3b-0.1-ft`) — optional TTS, more expressive, requires HF token + ZeroGPU |
| - **face-detection-retail-0004** + **person-detection-retail-0013** — OpenVINO IR, CPU, Intel OMZ |
|
|
| ## Social Media Post |
| - Youtube: https://www.youtube.com/@muflihma |
| - Instagram: https://www.instagram.com/0xcure |
