README.md

---
title: VAR Offside Visualizer
emoji: 🥅
colorFrom: purple
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---

# VAR-style Offside Visualizer

Reconstruct selected players in 3D from a **single broadcast clip** and draw a
VAR-style offside line you can rotate and inspect — no multi-camera rig, no pitch
calibration.

<img width="600" height="338" alt="EgyptIran (1)" src="https://github.com/user-attachments/assets/b5a096dc-6085-4f6a-b586-7bbb5150e11e" />
<img width="600" height="338" alt="Iran Belg" src="https://github.com/user-attachments/assets/9090e30e-32d7-43cc-b6eb-c0b1a1b8a53a" />

📝 **The story, the concept, and a walkthrough:**
[*Build a 3D Soccer Offside (VAR) System* — Roboflow blog](https://blog.roboflow.com/build-a-3d-soccer-offside-var-system/).
This README covers the setup, architecture, configuration, and technical detail the
blog intentionally leaves out.

---

## Pipeline (what each step actually does)

1. **Upload** a clip (`gr.Video`).
2. **Scrub** to the frame the ball is played — frames are seeked on demand (no bulk extract).
3. **Goal-parallel lines** on that frame: **✨ Auto-detect** (OpenCV: grass/white-line
   masks → Hough → vanishing-point RANSAC, proposes the two families; **↔ Flip** swaps
   them) or click 4 points by hand. These fix the offside axis via the vanishing point.
4. **Detect players (GPU)** with the selected detector — boxes + masks, cached per frame.
5. **Select** players by clicking their box/silhouette (click again to deselect).
6. **Mark defenders** (incl. GK). The offside line is drawn at the *furthest-forward*
   marked defender's furthest body point, **arms/hands excluded** (via MHR keypoints).
7. **Build** → reconstruct only the selected players, place them on a shared field frame,
   and render a Plotly scene with a draggable offside plane + OFFSIDE / NO-OFFSIDE verdict.
   Optionally **🎥 Generate a clean three.js scene** (gridlines, goal-direction arrow,
   in-scene verdict, **Save PNG**).

## Detectors (toggle in the UI)

| Option | Backend | Notes |
|---|---|---|
| **ViTDet (boxes)** | ViTDet-H Cascade Mask R-CNN (detectron2) | most accurate on broadcast footage; heavier |
| **RF-DETR (boxes)** | RF-DETR-Seg (Roboflow) | fast, Roboflow-native |
| **RF-DETR (segments)** | RF-DETR-Seg (Roboflow) | click silhouettes instead of boxes |

Each backend loads **lazily** — you only pay VRAM for the one you use. 3D reconstruction
is always **SAM 3D Body** (`facebook/sam-3d-body-dinov3`; DINOv3 backbone + MHR body model
+ MoGe2 FOV estimator).

## GPU / CPU boundary (the cost design)

The GPU is touched in **exactly two places** — detection and reconstruction — both in
`pipeline/gpu.py`, both cached per frame. Detection runs on the whole frame; the heavy
mesh reconstruction runs **only on the players you selected** (~3, not ~30). Everything
else (scrubbing, line geometry, placement, both renderers, the draggable plane) is pure
CPU on cached NumPy, so a dedicated GPU only ever does the heavy lifting.

## Code layout

```
app.py            Gradio UI + event wiring (CPU)
pipeline/
  video.py        frame seek / probe (CPU)
  gpu.py          detectors + SAM-3D reconstruction  ← the ONLY GPU code
  autolines.py    pitch-line detection + VP-RANSAC proposal (CPU, OpenCV)
  overlay.py      detection boxes / masks + line-click drawing (CPU)
  geometry.py     vanishing point, ground/up fit, field frame, offside, Plotly scene (CPU)
  threed.py       self-contained three.js scene (iframe srcdoc, CPU)
```

## Deploy (Docker SDK Space)

1. **Hardware:** a **GPU tier is required** (CPU fails at `.to("cuda")`). Comfortable
   minimum ≈ **L4 / A10G (24 GB)**; tested on **A100 (40 GB)**. VRAM is the limiter
   (detector + SAM-3D held together) — using RF-DETR and selecting few players lowers it.
2. **Secret `HF_TOKEN`:** a token for an account with **approved access to the gated
   `facebook/sam-3d-body-dinov3`**. Without it the weight download 401s.
3. First boot builds the image (compiles detectron2) and downloads ~7 GB of weights —
   give it time. The model then stays warm until you pause the Space.

**Cost control:** dedicated GPU bills continuously with no auto-shutoff — **pause the
Space** when not in use.

**Config env vars:** `HF_TOKEN` (required secret) · `SAM3D_REPO_ID`
(default `facebook/sam-3d-body-dinov3`) · `RFDETR_SIZE` (default `large`; `nano/small/medium`).

### Why not ZeroGPU?

ZeroGPU allocates the GPU per call, caps duration, enforces a daily quota, cold-loads the
~7 GB stack each time, and requires the Gradio SDK (not Docker) — all a poor fit for an
interactive scrubbing session.

## Accuracy & honesty

- Scale comes from **reconstructed body height**, so positions are approximate metres —
  good for relative offside ordering and a convincing visual, **not** sub-10 cm calls.
- **Level is onside** (offside law): any positive margin flags OFFSIDE, tagged **"(tight)"**
  when within the ±0.30 m band; orange = level / too-close-to-call on the onside side.
- Line detection **proposes** — you confirm/flip/redraw. A full metric homography was
  tried and rejected as the default (it can be confidently wrong on sparse frames).

## Explicitly later (see `TODO.md`)

Automatic pass-instant detection (ball tracking) · jersey/team auto-coloring · a
soccer-trained detector + field-keypoint homography (Roboflow) · three.js realism
(shadows / HDRI / GLB export).