# ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
📄 Paper •
🌐 Project Page •
🤗 Model •
📊 Dataset
---
## 🔥 ROMA's streaming understanding capabilities
> It supports proactive tasks, including event alerts and narration, alongside reactive question answering.
---
## 📖 Introduction
**ROMA** is a **real-time omni-multimodal assistant** for unified **streaming audio-video understanding**.
Unlike traditional VideoLLMs that only respond to explicit queries, ROMA supports both:
- **Reactive interaction** (question answering)
- **Proactive interaction** (event alerts and narration)
ROMA processes continuous streams as **synchronized multimodal units**, aligning dense audio signals with discrete video frames.
A lightweight **Speak Head** decouples **response timing** from **content generation**, enabling the model to autonomously decide **when to speak** in streaming environments.
The model is trained with a **two-stage streaming curriculum** and evaluated on a **unified benchmark suite covering 12 streaming tasks**.
---
## 🚀 Highlights
- **Unified Reactive + Proactive Streaming Interaction**
- **Streaming Audio-Video Understanding**
- **Lightweight Speak Head for Response Timing**
- **Two-Stage Streaming Curriculum Training**
- **Evaluation Across 12 Benchmarks**
---
## 📊 Performance
ROMA achieves **state-of-the-art performance on proactive streaming tasks** while remaining competitive on traditional reactive QA benchmarks.
### Proactive
### Reactive
### Omni-Modal Reactive
---
## 🧠 Model
| Model | Modalities | Capability |
|------|-----------|-----------|
| ROMA | Audio + Video + Text | Streaming Multimodal Understanding |
Model weights are available at:
👉 https://huggingface.co/EurekaTian/ROMA
---
## 📊 Dataset
The **ROMA Proactive Streaming Dataset** is released at:
👉 https://huggingface.co/datasets/EurekaTian/ROMA_proactive
Dataset statistics:
| Subset | Task | Samples |
|------|------|------|
| Event-Driven Alert | Proactive Monitoring | **27K** |
| Real-Time Narration | Streaming Captioning | **109K** |
| **Total** | | **136,193** |
Source datasets include:
- DiDeMo
- OOPS
- Charades-STA
- COIN
- YouCook2
- ActivityNet
These datasets are reformulated into **streaming interaction formats** for training proactive multimodal assistants.
---
## 📂 Repository Structure
```
ROMA
├── data
│ └── test_mix_data.json # example dataset format
├── eval # evaluation scripts
├── sh
│ └── train.sh # training entry
├── requirements.txt # dependencies
└── README.md
```
---
## 🏋️ Training
Install dependencies:
```bash
pip install -r requirements.txt
```
Run training:
```bash
bash sh/train.sh
```
Example data format:
```bash
data/test_mix_data.json
```
---
## 📈 Evaluation
All evaluation scripts are provided in:
```bash
eval/
```
The evaluation covers:
- Proactive Alert
- Real-Time Narration
- Reactive QA
---
## 📚 Citation
If you find this work useful, please cite:
```bibtex
@article{tian2026roma,
title={ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding},
author={Tian, Xueyun and Li, Wei and Xu, Bingbing and Dong, Heng and Wang, Yuanzhuo and Shen, Huawei},
journal={arXiv preprint arXiv:2601.10323},
year={2026}
}
```
---
## ⭐ Acknowledgement
If you find this repository helpful, please consider giving it a ⭐