README.md

metadata

title: VAR Offside Visualizer
emoji: 🥅
colorFrom: purple
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

VAR-style Offside Visualizer

Reconstruct selected players in 3D from a single broadcast clip and draw a VAR-style offside line you can rotate and inspect — no multi-camera rig, no pitch calibration.

📝 The story, the concept, and a walkthrough: Build a 3D Soccer Offside (VAR) System — Roboflow blog. This README covers the setup, architecture, configuration, and technical detail the blog intentionally leaves out.

---

Pipeline (what each step actually does)

1. Upload a clip (gr.Video).
2. Scrub to the frame the ball is played — frames are seeked on demand (no bulk extract).
3. Goal-parallel lines on that frame: ✨ Auto-detect (OpenCV: grass/white-line masks → Hough → vanishing-point RANSAC, proposes the two families; ↔ Flip swaps them) or click 4 points by hand. These fix the offside axis via the vanishing point.
4. Detect players (GPU) with the selected detector — boxes + masks, cached per frame.
5. Select players by clicking their box/silhouette (click again to deselect).
6. Mark defenders (incl. GK). The offside line is drawn at the furthest-forward marked defender's furthest body point, arms/hands excluded (via MHR keypoints).
7. Build → reconstruct only the selected players, place them on a shared field frame, and render a Plotly scene with a draggable offside plane + OFFSIDE / NO-OFFSIDE verdict. Optionally 🎥 Generate a clean three.js scene (gridlines, goal-direction arrow, in-scene verdict, Save PNG).

Detectors (toggle in the UI)

Option	Backend	Notes
ViTDet (boxes)	ViTDet-H Cascade Mask R-CNN (detectron2)	most accurate on broadcast footage; heavier
RF-DETR (boxes)	RF-DETR-Seg (Roboflow)	fast, Roboflow-native
RF-DETR (segments)	RF-DETR-Seg (Roboflow)	click silhouettes instead of boxes

Each backend loads lazily — you only pay VRAM for the one you use. 3D reconstruction is always SAM 3D Body (facebook/sam-3d-body-dinov3; DINOv3 backbone + MHR body model

• MoGe2 FOV estimator).

GPU / CPU boundary (the cost design)

The GPU is touched in exactly two places — detection and reconstruction — both in pipeline/gpu.py, both cached per frame. Detection runs on the whole frame; the heavy mesh reconstruction runs only on the players you selected (~3, not ~30). Everything else (scrubbing, line geometry, placement, both renderers, the draggable plane) is pure CPU on cached NumPy, so a dedicated GPU only ever does the heavy lifting.

Code layout

app.py            Gradio UI + event wiring (CPU)
pipeline/
  video.py        frame seek / probe (CPU)
  gpu.py          detectors + SAM-3D reconstruction  ← the ONLY GPU code
  autolines.py    pitch-line detection + VP-RANSAC proposal (CPU, OpenCV)
  overlay.py      detection boxes / masks + line-click drawing (CPU)
  geometry.py     vanishing point, ground/up fit, field frame, offside, Plotly scene (CPU)
  threed.py       self-contained three.js scene (iframe srcdoc, CPU)

Deploy (Docker SDK Space)

1. Hardware: a GPU tier is required (CPU fails at .to("cuda")). Comfortable minimum ≈ L4 / A10G (24 GB); tested on A100 (40 GB). VRAM is the limiter (detector + SAM-3D held together) — using RF-DETR and selecting few players lowers it.
2. Secret HF_TOKEN: a token for an account with approved access to the gated facebook/sam-3d-body-dinov3. Without it the weight download 401s.
3. First boot builds the image (compiles detectron2) and downloads ~7 GB of weights — give it time. The model then stays warm until you pause the Space.

Cost control: dedicated GPU bills continuously with no auto-shutoff — pause the Space when not in use.

Config env vars: HF_TOKEN (required secret) · SAM3D_REPO_ID (default facebook/sam-3d-body-dinov3) · RFDETR_SIZE (default large; nano/small/medium).

Why not ZeroGPU?

ZeroGPU allocates the GPU per call, caps duration, enforces a daily quota, cold-loads the ~7 GB stack each time, and requires the Gradio SDK (not Docker) — all a poor fit for an interactive scrubbing session.

Accuracy & honesty

• Scale comes from reconstructed body height, so positions are approximate metres — good for relative offside ordering and a convincing visual, not sub-10 cm calls.
• Level is onside (offside law): any positive margin flags OFFSIDE, tagged "(tight)" when within the ±0.30 m band; orange = level / too-close-to-call on the onside side.
• Line detection proposes — you confirm/flip/redraw. A full metric homography was tried and rejected as the default (it can be confidently wrong on sparse frames).

Explicitly later (see TODO.md)

Automatic pass-instant detection (ball tracking) · jersey/team auto-coloring · a soccer-trained detector + field-keypoint homography (Roboflow) · three.js realism (shadows / HDRI / GLB export).