README.md

metadata

title: Marquee
emoji: 🎙️
colorFrom: purple
colorTo: pink
sdk: gradio
sdk_version: 6.18.0
app_file: app.py
license: apache-2.0
short_description: Turn any clip into a broadcast starring your friends
tags:
  - build-small-hackathon
  - track:backyard
  - achievement:offgrid
  - achievement:offbrand

Marquee 🎙️

Drop a clip of friends doing anything. Marquee detects who's in it, you give each person a name, pick a commentary style, and an AI commentator calls the action play-by-play — by name, in sync — as a broadcast you can play back and export as an MP4.

What it does

1. Scan — Upload a clip (≤60s). OpenVINO detects and clusters people into tracks. You get a roster of face crops to name.
2. Generate — Qwen2.5-VL-7B watches key moments from the clip and writes commentary lines timed to the action, in whichever style you picked.
3. Voice — Each line gets spoken by a TTS model. Two options: Chatterbox (fast, runs on CPU) or Orpheus (slower, more expressive, requires GPU and a HuggingFace token).
4. Export — ffmpeg mixes the TTS audio over the original clip audio, burns in subtitles, and gives you a download.

Setup

Hardware

Must be ZeroGPU. Qwen2.5-VL-7B and Orpheus both need a GPU. Set it under Settings → Hardware → ZeroGPU in your Space. On a CPU-only Space the VLM is unusable.

Using Orpheus TTS

Orpheus uses canopylabs/orpheus-3b-0.1-ft, which is a gated model on HuggingFace. Two steps to unlock it:

1. Go to https://huggingface.co/canopylabs/orpheus-3b-0.1-ft and accept the access request. Takes about a minute to be approved.

2. Add your HuggingFace token as a Space secret (not a repo file):

   • Go to Settings → Variables and Secrets → New Secret
   • Name: HF_TOKEN
   • Value: a token from https://huggingface.co/settings/tokens with at least Read scope

Once the secret is set, restart the Space. The model will download on first use (~7GB). After that it's cached for the session.

Chatterbox (the default) needs no token and no extra setup.

Demo

https://youtu.be/f2I8m6ardkU

Local dev

pip install -r requirements.txt
python -c "from ov_models import download_models; download_models()"
python app.py
# → http://localhost:7860

Qwen lazy-loads on the first generate request. Orpheus lazy-loads on first TTS request with that model selected. Expect a slow first run.

For local Orpheus use you'll need to be logged in: huggingface-cli login.

How it's wired

The backend is a gradio.Server (a FastAPI subclass). This gives us ZeroGPU + the Gradio queue while letting custom routes take priority. The UI is a static HTML page served at /. The scan is a plain fetch() POST; commentary goes through the Gradio JS client so it hits the queue and ZeroGPU sees the @spaces.GPU decorator.

Marquee.html + marquee.css + marquee.js  →  served at "/"
  │
  ├─ fetch POST /api/scan      upload → normalize → detect+cluster → session + roster
  ├─ fetch GET  /video/{sid}   streams the normalized mp4
  ├─ gradio client /generate   key events → annotated frames → Qwen (ZeroGPU) → script
  └─ fetch POST /api/tts       script lines → Chatterbox or Orpheus (ZeroGPU) → WAV
  └─ fetch POST /api/export    WAVs + video → ffmpeg mix → MP4 download

Session state (video path, tracks, motion data) lives in a server-side dict keyed by session_id. Nothing heavy goes back and forth to the client between steps.

Files

File	What it does
app.py	FastAPI routes, session management, UI assembly
Marquee.html / marquee.css / marquee.js	The UI
commentary.py	Qwen2.5-VL personas, prompt, JSON parse
tts.py	Chatterbox + Orpheus generation, collision fix, WAV encoding
faces.py	OpenVINO detect → cluster, frame annotation
events.py	Motion-based key event selection
video.py	Rotation-aware ffmpeg normalize to 720p
ov_models.py	OpenVINO IR download + runtime wrappers

Models

• Qwen2.5-VL-7B-Instruct — commentary generation (ZeroGPU)
• Chatterbox 0.5B — default TTS, MIT license, CPU inference
• Orpheus 3B (canopylabs/orpheus-3b-0.1-ft) — optional TTS, more expressive, requires HF token + ZeroGPU
• face-detection-retail-0004 + person-detection-retail-0013 — OpenVINO IR, CPU, Intel OMZ

Social Media Post

• Youtube: https://www.youtube.com/@muflihma
• Instagram: https://www.instagram.com/0xcure