---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen2.5-Omni-7B
tags:
- multimodal
- tactile
- audio
- robotics
- active-perception
pipeline_tag: robotics
library_name: transformers
---
ROMA-7B
I saw, I touched, I understood.
This repository hosts the **model checkpoint** of **ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception**. The code, web demo, and usage instructions are in the [GitHub repository](https://github.com/GeWu-Lab/ROMA). The datasets (ROMI-2K and ROMA Bench) are released separately in [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA).
## Introduction
Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide **what** information is missing, **how** to acquire it, and **when** enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.
**ROMA** is an LLM-based system for **R**eal-World **O**bject-Centric **M**ulti-Sensory **A**ctive Perception. It integrates vision, audio, touch, and force into a *reasoning-interaction-feedback* loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.
**ROMA-7B** is the multi-sensory LLM at the core of this system. It is built on [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) and extended with:
- action / modality tokens for choosing interactions and sensory modalities;
- an audio branch (fine-tuned audio encoder and adapter) for contact sounds;
- an [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2) tactile branch (tactile encoder and adapter).
It is trained with multi-sensory alignment followed by active-perception SFT.
## Performance
Active perception task success rate on ROMA Bench (%). See the paper for the full results.
| Model | ROMA Bench (total) | Real-world execution (total) |
| :--- | :---: | :---: |
| GPT-5.4 | 43.5 | 40.2 |
| Gemini 3.5 Flash | 53.0 | 41.7 |
| Qwen 2.5-Omni | 22.0 | 23.5 |
| Qwen 3-Omni | 19.7 | 23.5 |
| **ROMA-7B** | **72.9** | **61.4** |
## Checkpoint Files
| File | Description |
| :--- | :--- |
| `model-0000*-of-00005.safetensors`, `model.safetensors.index.json` | Base Qwen2.5-Omni-7B weights |
| `anytouch2.pth`, `config-anytouch2.json` | AnyTouch 2 tactile encoder |
| `audio.bin` | Fine-tuned audio adapter and encoder |
| `tactile.bin` | Fine-tuned tactile adapter |
| `ROMA-LLM.bin` | ROMA LLM weights |
| `config.json`, `generation_config.json`, `preprocessor_config.json`, `chat_template.json`, `spk_dict.pt` | Model, generation, and processor configs |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json` | Tokenizer |
## Usage
ROMA-7B cannot be loaded with plain `transformers`: the tactile modality and the extra adapters are loaded by the code in the [GitHub repository](https://github.com/GeWu-Lab/ROMA), in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM).
1. Set up the environment following the [Installation](https://github.com/GeWu-Lab/ROMA#installation) section of the GitHub repository.
2. Download this checkpoint:
```bash
hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B
```
The demo reads the checkpoint directory from `MODEL_PATH_2_5` in `config/global_name.py`, which defaults to `resources/models/ROMA-7B`. Edit it if you download the checkpoint elsewhere.
3. Launch the web demo:
```bash
python demo.py --gpu 0 --port 7860
```
Then open `http://localhost:7860`, type a question (or pick an example), and press **Ask ROMA**.
The full checkpoint is about 39 GB on disk.
## Related Resources
- **ROMI-2K** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
- **ROMA Bench** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.
The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.
## License
This model is released under the Apache 2.0 license (see `LICENSE`), following its base model Qwen2.5-Omni-7B.
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{feng2026roma,
title = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
author = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
Wang, Zhongyuan and Yao, Guocai and Hu, Di},
journal = {arXiv preprint arXiv:2610.06955},
url = {https://arxiv.org/abs/2610.06955},
year = {2026}
}
```
## Acknowledgements
ROMA builds on [Qwen2.5-Omni](https://github.com/QwenLM/Qwen2.5-Omni), [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2), and [AnyGrasp](https://github.com/graspnet/anygrasp_sdk). We thank the authors for their open-source contributions.