File size: 6,429 Bytes
49977e7 a5d4c42 49977e7 2e8795f 49977e7 2e8795f 49977e7 2e8795f 49977e7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen2.5-Omni-7B
tags:
- multimodal
- tactile
- audio
- robotics
- active-perception
pipeline_tag: robotics
library_name: transformers
---
<p align="center">
<img src="assets/logo.png" width="200" alt="ROMA logo">
</p>
<h1 align="center">ROMA-7B</h1>
<h3 align="center"><em>I saw, I touched, I understood.</em></h3>
<p align="center">
<a href="https://gewu-lab.github.io/ROMA/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
<a href="https://arxiv.org/abs/2610.06955"><img src="https://img.shields.io/badge/arXiv-2610.06955-b31b1b" alt="Paper"></a>
<a href="https://github.com/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"></a>
<a href="https://huggingface.co/datasets/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow" alt="Hugging Face Dataset"></a>
</p>
This repository hosts the **model checkpoint** of **ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception**. The code, web demo, and usage instructions are in the [GitHub repository](https://github.com/GeWu-Lab/ROMA). The datasets (ROMI-2K and ROMA Bench) are released separately in [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA).
## Introduction
<p align="center">
<img src="assets/teaser.png" width="90%" alt="ROMA: LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception">
</p>
Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide **what** information is missing, **how** to acquire it, and **when** enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.
**ROMA** is an LLM-based system for **R**eal-World **O**bject-Centric **M**ulti-Sensory **A**ctive Perception. It integrates vision, audio, touch, and force into a *reasoning-interaction-feedback* loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.
**ROMA-7B** is the multi-sensory LLM at the core of this system. It is built on [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) and extended with:
- action / modality tokens for choosing interactions and sensory modalities;
- an audio branch (fine-tuned audio encoder and adapter) for contact sounds;
- an [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2) tactile branch (tactile encoder and adapter).
It is trained with multi-sensory alignment followed by active-perception SFT.
## Performance
Active perception task success rate on ROMA Bench (%). See the paper for the full results.
| Model | ROMA Bench (total) | Real-world execution (total) |
| :--- | :---: | :---: |
| GPT-5.4 | 43.5 | 40.2 |
| Gemini 3.5 Flash | 53.0 | 41.7 |
| Qwen 2.5-Omni | 22.0 | 23.5 |
| Qwen 3-Omni | 19.7 | 23.5 |
| **ROMA-7B** | **72.9** | **61.4** |
## Checkpoint Files
| File | Description |
| :--- | :--- |
| `model-0000*-of-00005.safetensors`, `model.safetensors.index.json` | Base Qwen2.5-Omni-7B weights |
| `anytouch2.pth`, `config-anytouch2.json` | AnyTouch 2 tactile encoder |
| `audio.bin` | Fine-tuned audio adapter and encoder |
| `tactile.bin` | Fine-tuned tactile adapter |
| `ROMA-LLM.bin` | ROMA LLM weights |
| `config.json`, `generation_config.json`, `preprocessor_config.json`, `chat_template.json`, `spk_dict.pt` | Model, generation, and processor configs |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json` | Tokenizer |
## Usage
ROMA-7B cannot be loaded with plain `transformers`: the tactile modality and the extra adapters are loaded by the code in the [GitHub repository](https://github.com/GeWu-Lab/ROMA), in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM).
1. Set up the environment following the [Installation](https://github.com/GeWu-Lab/ROMA#installation) section of the GitHub repository.
2. Download this checkpoint:
```bash
hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B
```
The demo reads the checkpoint directory from `MODEL_PATH_2_5` in `config/global_name.py`, which defaults to `resources/models/ROMA-7B`. Edit it if you download the checkpoint elsewhere.
3. Launch the web demo:
```bash
python demo.py --gpu 0 --port 7860
```
Then open `http://localhost:7860`, type a question (or pick an example), and press **Ask ROMA**.
The full checkpoint is about 39 GB on disk.
## Related Resources
- **ROMI-2K** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
- **ROMA Bench** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.
The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.
## License
This model is released under the Apache 2.0 license (see `LICENSE`), following its base model Qwen2.5-Omni-7B.
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{feng2026roma,
title = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
author = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
Wang, Zhongyuan and Yao, Guocai and Hu, Di},
journal = {arXiv preprint arXiv:2610.06955},
url = {https://arxiv.org/abs/2610.06955},
year = {2026}
}
```
## Acknowledgements
ROMA builds on [Qwen2.5-Omni](https://github.com/QwenLM/Qwen2.5-Omni), [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2), and [AnyGrasp](https://github.com/graspnet/anygrasp_sdk). We thank the authors for their open-source contributions.
|