README.md

metadata

title: Marquee
emoji: 🎙️
colorFrom: purple
colorTo: pink
sdk: gradio
sdk_version: 6.18.0
app_file: app.py
license: apache-2.0
short_description: Turn any clip into a broadcast starring your friends
tags:
  - build-small-hackathon

Marquee 🎙️ — Everyone's a headliner.

Drop a clip of friends doing anything. Marquee recognizes who's in it, you give each person a stage name, pick a show style, and an AI commentator calls the action play-by-play, by name, in sync — as a broadcast you can play and export.

⚙️ Running it on a Space (read this first)

• Hardware must be ZeroGPU. The commentary model runs inside @spaces.GPU, which only has a GPU on Nvidia ZeroGPU hardware. On a plain CPU Space the 7B VLM has no GPU and is effectively unusable. Set it in Settings → Hardware → ZeroGPU.
• No secrets needed at runtime. Everything downloads from public sources (Qwen from the HF Hub, the face models from Intel's OMZ storage). Do not commit any HF_TOKEN/.env — if you need one for gradio deploy, keep it local and add it as a Space secret, never a file in the repo.
• Nothing to pre-download. download_models() fetches the three tiny OpenVINO IR files at first boot (~5 MB total). First boot also warms up Qwen.

How it's wired (the important part)

The backend is a gradio.Server (a FastAPI subclass — Gradio 6). It gives us ZeroGPU + the gradio runtime while letting our own routes take priority. The custom UI is a static HTML page at /. The CPU-only scan is a plain fetch() route; the GPU commentary is an @app.api() endpoint called through the Gradio JS client, which is what wires it into the queue and lets ZeroGPU detect the @spaces.GPU function. No iframe, no SSR fighting us for /.

Static UI (Marquee.html + marquee.css + marquee.js)  ──►  served at "/"
   │  fetch() for scan · Gradio JS client for generate
   ▼
FastAPI (app.py)
   ├─ POST /api/scan      (file)                -> normalize 720p -> detect+cluster faces
   │                                               -> session + roster (face crops as data URLs)
   ├─ GET  /video/{sid}                         -> streams the normalized mp4
   └─ @app.api /generate  (session, names, vibe)-> key events -> name-grounded frames
                                                   -> Qwen2.5-VL (ZeroGPU) -> {t,text} script

Heavy per-session state (identities, motion, normalized video path) lives in a server-side SESSIONS dict keyed by the session_id the scan call returns, so the two stateless client calls never round-trip embeddings.

Face recognition is used only to feed names to the VLM — boxes are never shown. The broadcast player reads the real normalized <video> and syncs the chyron + play-by-play rail to video.currentTime.

Files

• app.py — FastAPI backend + the two API endpoints + UI assembly
• Marquee.html / marquee.css / marquee.js — the custom UI (design + flow)
• video.py — rotation-aware 720p normalize (one upright source of truth)
• faces.py — OpenVINO detect → align → embed → cluster (CPU)
• events.py — motion-based key events
• commentary.py — Qwen2.5-VL personas (incl. Diva Hour) + strict JSON parse
• ov_models.py — OpenVINO IR auto-download + thin runtime wrappers

Models (all open)

• face-detection-retail-0004, landmarks-regression-retail-0009, face-reidentification-retail-0095 (OpenVINO IR, CPU)
• Qwen2.5-VL-7B-Instruct (ZeroGPU)

North star (next iteration)

• TTS per line (hyped delivery) mixed under the clip's own audio.
• Rendered MP4 export with captions (and voice) burned in.
• Canvas hype FX on key moments (data's already there).

Local dev

pip install -r requirements.txt
python -c "from ov_models import download_models; download_models()"
python app.py    # http://localhost:7860  (Qwen lazy-loads on first generate)