Running ROMA reproducibly on Quadro RTX 6000 (×4)

This guide reproduces the inference / real-time streaming demos of ROMA (arXiv:2601.10323) on a workstation with 4× Quadro RTX 6000 (Turing, sm_75, x86_64, 24 GB each = 96 GB), driver 570.x / CUDA 12.8, using Docker for reproducibility. It covers the real-time proactive demo (the Speak Head decides when to respond), plus narration and reactive QA. See ARCHITECTURE.md for how ROMA works.

Training and the full evaluation suite are out of scope for this image.

Why a Turing-specific image?

The Quadro RTX 6000 is Turing (sm_75), which changes three things vs. the repo's defaults:

Repo default	Problem on Turing	This image
`attn_implementation="flash_attention_2"`	FlashAttention-2 needs Ampere (sm_80+)	defaults to `sdpa` (`ROMA_ATTN`)
`torch_dtype=torch.bfloat16`	bf16 not hardware-accelerated on Turing	defaults to fp16 (`ROMA_DTYPE`)
`flash_attn==2.7.4.post1` in requirements	no sm_75 wheel; source build fails	removed from `requirements-rtx6000.txt`
single-GPU fp16 (~22 GB)	won't fit 24 GB + KV-cache	sharded across all 4 GPUs via `device_map="auto"`

x86_64 means the original +cu124 PyTorch wheels are valid, so this image stays close to the upstream pins (unlike an ARM/GH200 build). We base on nvidia/cuda:12.4.1-cudnn-runtime and install official cu124 PyTorch wheels, which include Turing kernels.

Prerequisites

NVIDIA driver (present: 570.195.03) + NVIDIA Container Toolkit. Verify: docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi lists all 4 cards.
Docker with Compose v2, ~40 GB free disk (image + checkpoint).
Network access to pypi.org, download.pytorch.org, github.com, huggingface.co.

1. Build

git clone <your-repo-url> ROMA && cd ROMA
docker compose -f docker/docker-rtx6000/docker-compose.yml build

2. Start an interactive container (all 4 GPUs)

docker compose -f docker/docker-rtx6000/docker-compose.yml run --rm --service-ports roma bash

The checkpoint, HF cache and demo media are bind-mounted to the host (./whole_model, ./hf_cache, ./demo_media) so they persist across runs.

3. Download the checkpoint (inside the container)

bash scripts/rtx6000/download_model.sh        # -> whole_model/model  (~16-22 GB)

(If the pull is rate-limited/gated, set HF_TOKEN.)

4. Run a real-time demo (inside the container)

bash scripts/rtx6000/run_demo.sh proactive    # real-time proactive event alert (default)
# or
bash scripts/rtx6000/run_demo.sh narration    # real-time streaming narration
bash scripts/rtx6000/run_demo.sh mme          # reactive multimodal QA

Open http://<host>:7860 and click ▶ Start Detection Stream. ROMA streams its output live, with the Speak Head triggering above its threshold.

Using your own clips

ROMA_VIDEO=/app/demo_media/my_clip.mp4 \
ROMA_AUDIO=/app/demo_media/my_clip.wav \
bash scripts/rtx6000/run_demo.sh proactive

A bundled sample video: gradio/aCkbw-aI4xU_cut80s.mp4 (the narration default).

Smoke test (verify the environment)

Inside the container:

python -c "import torch; print(torch.__version__, torch.cuda.device_count(), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))"
# expect: 2.6.0+cu124  4  'Quadro RTX 6000'  (7, 5)
python -c "from transformers import Qwen2_5OmniModel; print('Qwen2_5OmniModel OK')"

While a demo runs, nvidia-smi should show the model sharded across GPUs 0–3 (~5–6 GB each).

Fallback matrix (env vars)

Symptom	Try
`sdpa` rejected by the transformers fork	`ROMA_ATTN=eager bash scripts/rtx6000/run_demo.sh ...`
Gate probabilities print as `nan` (fp16 overflow)	`ROMA_DTYPE=bfloat16 ...` (bf16 runs in software on Turing — slower but safe)
`device_map="auto"` mis-places modules / cross-GPU error	run on one card with 8-bit: `CUDA_VISIBLE_DEVICES=0 ROMA_LOAD_8BIT=1 ...`
Want to free 3 GPUs for other work	`CUDA_VISIBLE_DEVICES=0 ROMA_LOAD_8BIT=1 ...` (fits ~10–12 GB on one card)
8-bit load errors on the omni encoders	edit `llm_int8_skip_modules` in the demo loader, or set `ROMA_LOAD_8BIT=0`

Notes

The demos only use model.thinker; the loader calls disable_talker() (best-effort) to free memory.
attn_implementation and dtype are read from env at load time, so the same scripts also run with the paper's original settings on an Ampere+ GPU: ROMA_ATTN=flash_attention_2 ROMA_DTYPE=bfloat16.