README-RTX6000.md

# Running ROMA reproducibly on Quadro RTX 6000 (×4)

This guide reproduces the **inference / real-time streaming demos** of
[ROMA (arXiv:2601.10323)](https://arxiv.org/abs/2601.10323) on a workstation with
**4× Quadro RTX 6000** (Turing, sm_75, x86_64, 24 GB each = 96 GB), driver 570.x / CUDA 12.8, using
Docker for reproducibility. It covers the **real-time proactive** demo (the Speak Head decides *when*
to respond), plus narration and reactive QA. See [ARCHITECTURE.md](ARCHITECTURE.md) for how ROMA works.

> Training and the full evaluation suite are **out of scope** for this image.

## Why a Turing-specific image?

The Quadro RTX 6000 is **Turing (sm_75)**, which changes three things vs. the repo's defaults:

| Repo default | Problem on Turing | This image |
|---|---|---|
| `attn_implementation="flash_attention_2"` | FlashAttention-2 needs Ampere (sm_80+) | defaults to **`sdpa`** (`ROMA_ATTN`) |
| `torch_dtype=torch.bfloat16` | bf16 not hardware-accelerated on Turing | defaults to **fp16** (`ROMA_DTYPE`) |
| `flash_attn==2.7.4.post1` in requirements | no sm_75 wheel; source build fails | **removed** from `requirements-rtx6000.txt` |
| single-GPU fp16 (~22 GB) | won't fit 24 GB + KV-cache | **sharded across all 4 GPUs** via `device_map="auto"` |

x86_64 means the original `+cu124` PyTorch wheels are valid, so this image stays close to the
upstream pins (unlike an ARM/GH200 build). We base on `nvidia/cuda:12.4.1-cudnn-runtime` and install
**official cu124 PyTorch wheels**, which include Turing kernels.

## Prerequisites

- NVIDIA driver (present: 570.195.03) + **NVIDIA Container Toolkit**.
  Verify: `docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi` lists all 4 cards.
- Docker with Compose v2, ~40 GB free disk (image + checkpoint).
- Network access to `pypi.org`, `download.pytorch.org`, `github.com`, `huggingface.co`.

## 1. Build

```bash
git clone <your-repo-url> ROMA && cd ROMA
docker compose -f docker/docker-rtx6000/docker-compose.yml build
```

## 2. Start an interactive container (all 4 GPUs)

```bash
docker compose -f docker/docker-rtx6000/docker-compose.yml run --rm --service-ports roma bash
```

The checkpoint, HF cache and demo media are bind-mounted to the host (`./whole_model`, `./hf_cache`,
`./demo_media`) so they persist across runs.

## 3. Download the checkpoint (inside the container)

```bash
bash scripts/rtx6000/download_model.sh        # -> whole_model/model  (~16-22 GB)
```

(If the pull is rate-limited/gated, set `HF_TOKEN`.)

## 4. Run a real-time demo (inside the container)

```bash
bash scripts/rtx6000/run_demo.sh proactive    # real-time proactive event alert (default)
# or
bash scripts/rtx6000/run_demo.sh narration    # real-time streaming narration
bash scripts/rtx6000/run_demo.sh mme          # reactive multimodal QA
```

Open **`http://<host>:7860`** and click **▶ Start Detection Stream**. ROMA streams its output live,
with the Speak Head triggering above its threshold.

### Using your own clips

```bash
ROMA_VIDEO=/app/demo_media/my_clip.mp4 \
ROMA_AUDIO=/app/demo_media/my_clip.wav \
bash scripts/rtx6000/run_demo.sh proactive
```

A bundled sample video: `gradio/aCkbw-aI4xU_cut80s.mp4` (the `narration` default).

## Smoke test (verify the environment)

Inside the container:

```bash
python -c "import torch; print(torch.__version__, torch.cuda.device_count(), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))"
# expect: 2.6.0+cu124  4  'Quadro RTX 6000'  (7, 5)
python -c "from transformers import Qwen2_5OmniModel; print('Qwen2_5OmniModel OK')"
```

While a demo runs, `nvidia-smi` should show the model **sharded across GPUs 0–3** (~5–6 GB each).

## Fallback matrix (env vars)

| Symptom | Try |
|---|---|
| `sdpa` rejected by the transformers fork | `ROMA_ATTN=eager bash scripts/rtx6000/run_demo.sh ...` |
| Gate probabilities print as `nan` (fp16 overflow) | `ROMA_DTYPE=bfloat16 ...` (bf16 runs in software on Turing — slower but safe) |
| `device_map="auto"` mis-places modules / cross-GPU error | run on one card with 8-bit: `CUDA_VISIBLE_DEVICES=0 ROMA_LOAD_8BIT=1 ...` |
| Want to free 3 GPUs for other work | `CUDA_VISIBLE_DEVICES=0 ROMA_LOAD_8BIT=1 ...` (fits ~10–12 GB on one card) |
| 8-bit load errors on the omni encoders | edit `llm_int8_skip_modules` in the demo loader, or set `ROMA_LOAD_8BIT=0` |

## Notes

- The demos only use `model.thinker`; the loader calls `disable_talker()` (best-effort) to free memory.
- `attn_implementation` and dtype are read from env at load time, so the same scripts also run with the
  paper's original settings on an Ampere+ GPU: `ROMA_ATTN=flash_attention_2 ROMA_DTYPE=bfloat16`.