readme.md

ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding

📄 Paper • 🌐 Project Page • 🤗 Model • 📊 Dataset

---

🔥 ROMA's streaming understanding capabilities

It supports proactive tasks, including event alerts and narration, alongside reactive question answering.

---

📖 Introduction

ROMA is a real-time omni-multimodal assistant for unified streaming audio-video understanding.

Unlike traditional VideoLLMs that only respond to explicit queries, ROMA supports both:

• Reactive interaction (question answering)
• Proactive interaction (event alerts and narration)

ROMA processes continuous streams as synchronized multimodal units, aligning dense audio signals with discrete video frames.
A lightweight Speak Head decouples response timing from content generation, enabling the model to autonomously decide when to speak in streaming environments.

The model is trained with a two-stage streaming curriculum and evaluated on a unified benchmark suite covering 12 streaming tasks.

---

🚀 Highlights

• Unified Reactive + Proactive Streaming Interaction
• Streaming Audio-Video Understanding
• Lightweight Speak Head for Response Timing
• Two-Stage Streaming Curriculum Training
• Evaluation Across 12 Benchmarks

---

📊 Performance

ROMA achieves state-of-the-art performance on proactive streaming tasks while remaining competitive on traditional reactive QA benchmarks.

Proactive

Reactive

Omni-Modal Reactive

---

🧠 Model

Model	Modalities	Capability
ROMA	Audio + Video + Text	Streaming Multimodal Understanding

Model weights are available at:

👉 https://huggingface.co/EurekaTian/ROMA

---

📊 Dataset

The ROMA Proactive Streaming Dataset is released at:

👉 https://huggingface.co/datasets/EurekaTian/ROMA_proactive

Dataset statistics:

Subset	Task	Samples
Event-Driven Alert	Proactive Monitoring	27K
Real-Time Narration	Streaming Captioning	109K
Total		136,193

Source datasets include:

• DiDeMo
• OOPS
• Charades-STA
• COIN
• YouCook2
• ActivityNet

These datasets are reformulated into streaming interaction formats for training proactive multimodal assistants.

---

📂 Repository Structure

ROMA
├── data
│   └── test_mix_data.json        # example dataset format
├── eval                          # evaluation scripts
├── sh
│   └── train.sh                  # training entry
├── requirements.txt              # dependencies
└── README.md

---

🏋️ Training

Install dependencies:

pip install -r requirements.txt

Run training:

bash sh/train.sh

Example data format:

data/test_mix_data.json

---

📈 Evaluation

All evaluation scripts are provided in:

eval/

The evaluation covers:

• Proactive Alert
• Real-Time Narration
• Reactive QA

---

📚 Citation

If you find this work useful, please cite:

@article{tian2026roma,
  title={ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding},
  author={Tian, Xueyun and Li, Wei and Xu, Bingbing and Dong, Heng and Wang, Yuanzhuo and Shen, Huawei},
  journal={arXiv preprint arXiv:2601.10323},
  year={2026}
}

---

⭐ Acknowledgement

If you find this repository helpful, please consider giving it a ⭐