ROMA-7B / README.md

Upload README.md with huggingface_hub

a5d4c42 verified 4 days ago

6.43 kB

---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen2.5-Omni-7B
tags:
- multimodal
- tactile
- audio
- robotics
- active-perception
pipeline_tag: robotics
library_name: transformers
---
<p align="center">
  <img src="assets/logo.png" width="200" alt="ROMA logo">
</p>

<h1 align="center">ROMA-7B</h1>

<h3 align="center"><em>I saw, I touched, I understood.</em></h3>

<p align="center">
  <a href="https://gewu-lab.github.io/ROMA/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
  <a href="https://arxiv.org/abs/2610.06955"><img src="https://img.shields.io/badge/arXiv-2610.06955-b31b1b" alt="Paper"></a>
  <a href="https://github.com/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"></a>
  <a href="https://huggingface.co/datasets/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow" alt="Hugging Face Dataset"></a>
</p>

This repository hosts the **model checkpoint** of **ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception**. The code, web demo, and usage instructions are in the [GitHub repository](https://github.com/GeWu-Lab/ROMA). The datasets (ROMI-2K and ROMA Bench) are released separately in [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA).

## Introduction

<p align="center">
  <img src="assets/teaser.png" width="90%" alt="ROMA: LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception">
</p>

Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide **what** information is missing, **how** to acquire it, and **when** enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.

**ROMA** is an LLM-based system for **R**eal-World **O**bject-Centric **M**ulti-Sensory **A**ctive Perception. It integrates vision, audio, touch, and force into a *reasoning-interaction-feedback* loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.

**ROMA-7B** is the multi-sensory LLM at the core of this system. It is built on [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) and extended with:

- action / modality tokens for choosing interactions and sensory modalities;
- an audio branch (fine-tuned audio encoder and adapter) for contact sounds;
- an [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2) tactile branch (tactile encoder and adapter).

It is trained with multi-sensory alignment followed by active-perception SFT.

## Performance

Active perception task success rate on ROMA Bench (%). See the paper for the full results.

| Model | ROMA Bench (total) | Real-world execution (total) |
| :--- | :---: | :---: |
| GPT-5.4 | 43.5 | 40.2 |
| Gemini 3.5 Flash | 53.0 | 41.7 |
| Qwen 2.5-Omni | 22.0 | 23.5 |
| Qwen 3-Omni | 19.7 | 23.5 |
| **ROMA-7B** | **72.9** | **61.4** |

## Checkpoint Files

| File | Description |
| :--- | :--- |
| `model-0000*-of-00005.safetensors`, `model.safetensors.index.json` | Base Qwen2.5-Omni-7B weights |
| `anytouch2.pth`, `config-anytouch2.json` | AnyTouch 2 tactile encoder |
| `audio.bin` | Fine-tuned audio adapter and encoder |
| `tactile.bin` | Fine-tuned tactile adapter |
| `ROMA-LLM.bin` | ROMA LLM weights |
| `config.json`, `generation_config.json`, `preprocessor_config.json`, `chat_template.json`, `spk_dict.pt` | Model, generation, and processor configs |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json` | Tokenizer |

## Usage

ROMA-7B cannot be loaded with plain `transformers`: the tactile modality and the extra adapters are loaded by the code in the [GitHub repository](https://github.com/GeWu-Lab/ROMA), in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM).

1. Set up the environment following the [Installation](https://github.com/GeWu-Lab/ROMA#installation) section of the GitHub repository.

2. Download this checkpoint:

   ```bash
   hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B
   ```

   The demo reads the checkpoint directory from `MODEL_PATH_2_5` in `config/global_name.py`, which defaults to `resources/models/ROMA-7B`. Edit it if you download the checkpoint elsewhere.

3. Launch the web demo:

   ```bash
   python demo.py --gpu 0 --port 7860
   ```

   Then open `http://localhost:7860`, type a question (or pick an example), and press **Ask ROMA**.

The full checkpoint is about 39 GB on disk.

## Related Resources

- **ROMI-2K** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
- **ROMA Bench** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.

The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.

## License

This model is released under the Apache 2.0 license (see `LICENSE`), following its base model Qwen2.5-Omni-7B.

## Citation

If you find this work useful, please consider citing:

```bibtex
@article{feng2026roma,
  title   = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
  author  = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
             Wang, Zhongyuan and Yao, Guocai and Hu, Di},
  journal = {arXiv preprint arXiv:2610.06955},
  url     = {https://arxiv.org/abs/2610.06955},
  year    = {2026}
}
```

## Acknowledgements

ROMA builds on [Qwen2.5-Omni](https://github.com/QwenLM/Qwen2.5-Omni), [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2), and [AnyGrasp](https://github.com/graspnet/anygrasp_sdk). We thank the authors for their open-source contributions.

	---
	license: apache-2.0
	language:
	- en
	base_model:
	- Qwen/Qwen2.5-Omni-7B
	tags:
	- multimodal
	- tactile
	- audio
	- robotics
	- active-perception
	pipeline_tag: robotics
	library_name: transformers
	---

	<p align="center">
	<img src="assets/logo.png" width="200" alt="ROMA logo">
	</p>

	<h1 align="center">ROMA-7B</h1>

	<h3 align="center"><em>I saw, I touched, I understood.</em></h3>

	<p align="center">
	<a href="https://gewu-lab.github.io/ROMA/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
	<a href="https://arxiv.org/abs/2610.06955"><img src="https://img.shields.io/badge/arXiv-2610.06955-b31b1b" alt="Paper"></a>
	<a href="https://github.com/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/Code-GitHub-black" alt="Code"></a>
	<a href="https://huggingface.co/datasets/GeWu-Lab/ROMA"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow" alt="Hugging Face Dataset"></a>
	</p>

	This repository hosts the model checkpoint of ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception. The code, web demo, and usage instructions are in the [GitHub repository](https://github.com/GeWu-Lab/ROMA). The datasets (ROMI-2K and ROMA Bench) are released separately in [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA).

	## Introduction

	<p align="center">
	<img src="assets/teaser.png" width="90%" alt="ROMA: LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception">
	</p>

	Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide what information is missing, how to acquire it, and when enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.

	ROMA is an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. It integrates vision, audio, touch, and force into a reasoning-interaction-feedback loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.

	ROMA-7B is the multi-sensory LLM at the core of this system. It is built on [Qwen2.5-Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) and extended with:

	- action / modality tokens for choosing interactions and sensory modalities;
	- an audio branch (fine-tuned audio encoder and adapter) for contact sounds;
	- an [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2) tactile branch (tactile encoder and adapter).

	It is trained with multi-sensory alignment followed by active-perception SFT.

	## Performance

	Active perception task success rate on ROMA Bench (%). See the paper for the full results.

	\| Model \| ROMA Bench (total) \| Real-world execution (total) \|
	\| :--- \| :---: \| :---: \|
	\| GPT-5.4 \| 43.5 \| 40.2 \|
	\| Gemini 3.5 Flash \| 53.0 \| 41.7 \|
	\| Qwen 2.5-Omni \| 22.0 \| 23.5 \|
	\| Qwen 3-Omni \| 19.7 \| 23.5 \|
	\| ROMA-7B \| 72.9 \| 61.4 \|

	## Checkpoint Files

	\| File \| Description \|
	\| :--- \| :--- \|
	\| `model-0000*-of-00005.safetensors`, `model.safetensors.index.json` \| Base Qwen2.5-Omni-7B weights \|
	\| `anytouch2.pth`, `config-anytouch2.json` \| AnyTouch 2 tactile encoder \|
	\| `audio.bin` \| Fine-tuned audio adapter and encoder \|
	\| `tactile.bin` \| Fine-tuned tactile adapter \|
	\| `ROMA-LLM.bin` \| ROMA LLM weights \|
	\| `config.json`, `generation_config.json`, `preprocessor_config.json`, `chat_template.json`, `spk_dict.pt` \| Model, generation, and processor configs \|
	\| `tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json` \| Tokenizer \|

	## Usage

	ROMA-7B cannot be loaded with plain `transformers`: the tactile modality and the extra adapters are loaded by the code in the [GitHub repository](https://github.com/GeWu-Lab/ROMA), in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM).

	1. Set up the environment following the [Installation](https://github.com/GeWu-Lab/ROMA#installation) section of the GitHub repository.

	2. Download this checkpoint:

	```bash
	hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B
	```

	The demo reads the checkpoint directory from `MODEL_PATH_2_5` in `config/global_name.py`, which defaults to `resources/models/ROMA-7B`. Edit it if you download the checkpoint elsewhere.

	3. Launch the web demo:

	```bash
	python demo.py --gpu 0 --port 7860
	```

	Then open `http://localhost:7860`, type a question (or pick an example), and press Ask ROMA.

	The full checkpoint is about 39 GB on disk.

	## Related Resources

	- ROMI-2K ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
	- ROMA Bench ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.

	The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.

	## License

	This model is released under the Apache 2.0 license (see `LICENSE`), following its base model Qwen2.5-Omni-7B.

	## Citation

	If you find this work useful, please consider citing:

	```bibtex
	@article{feng2026roma,
	title = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
	author = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
	Wang, Zhongyuan and Yao, Guocai and Hu, Di},
	journal = {arXiv preprint arXiv:2610.06955},
	url = {https://arxiv.org/abs/2610.06955},
	year = {2026}
	}
	```

	## Acknowledgements

	ROMA builds on [Qwen2.5-Omni](https://github.com/QwenLM/Qwen2.5-Omni), [AnyTouch 2](https://github.com/GeWu-Lab/AnyTouch2), and [AnyGrasp](https://github.com/graspnet/anygrasp_sdk). We thank the authors for their open-source contributions.