Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -13,6 +13,10 @@ tags:
|
|
| 13 |
pipeline_tag: text-generation
|
| 14 |
---
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
<h1 align="center">ROMA-7B</h1>
|
| 17 |
|
| 18 |
<h3 align="center"><em>I saw, I touched, I understood.</em></h3>
|
|
@@ -28,6 +32,10 @@ This repository hosts the **model checkpoint** of **ROMA: LLM System for Real-Wo
|
|
| 28 |
|
| 29 |
## Introduction
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide **what** information is missing, **how** to acquire it, and **when** enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.
|
| 32 |
|
| 33 |
**ROMA** is an LLM-based system for **R**eal-World **O**bject-Centric **M**ulti-Sensory **A**ctive Perception. It integrates vision, audio, touch, and force into a *reasoning-interaction-feedback* loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.
|
|
@@ -91,7 +99,7 @@ The full checkpoint is about 39 GB on disk.
|
|
| 91 |
## Related Resources
|
| 92 |
|
| 93 |
- **ROMI-2K** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
|
| 94 |
-
- **ROMA Bench**: 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.
|
| 95 |
|
| 96 |
The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.
|
| 97 |
|
|
|
|
| 13 |
pipeline_tag: text-generation
|
| 14 |
---
|
| 15 |
|
| 16 |
+
<p align="center">
|
| 17 |
+
<img src="assets/logo.png" width="200" alt="ROMA logo">
|
| 18 |
+
</p>
|
| 19 |
+
|
| 20 |
<h1 align="center">ROMA-7B</h1>
|
| 21 |
|
| 22 |
<h3 align="center"><em>I saw, I touched, I understood.</em></h3>
|
|
|
|
| 32 |
|
| 33 |
## Introduction
|
| 34 |
|
| 35 |
+
<p align="center">
|
| 36 |
+
<img src="assets/teaser.png" width="90%" alt="ROMA: LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception">
|
| 37 |
+
</p>
|
| 38 |
+
|
| 39 |
Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide **what** information is missing, **how** to acquire it, and **when** enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.
|
| 40 |
|
| 41 |
**ROMA** is an LLM-based system for **R**eal-World **O**bject-Centric **M**ulti-Sensory **A**ctive Perception. It integrates vision, audio, touch, and force into a *reasoning-interaction-feedback* loop: the LLM identifies the missing evidence and selects the target object, the interaction (`lift`, `press`, `collide`, `shake`, `rotate`, `squeeze`), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.
|
|
|
|
| 99 |
## Related Resources
|
| 100 |
|
| 101 |
- **ROMI-2K** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
|
| 102 |
+
- **ROMA Bench** ([dataset](https://huggingface.co/datasets/GeWu-Lab/ROMA)): 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.
|
| 103 |
|
| 104 |
The datasets are hosted in a separate repository, [GeWu-Lab/ROMA](https://huggingface.co/datasets/GeWu-Lab/ROMA), and are not included in this model repository. See the [project page](https://gewu-lab.github.io/ROMA/) and the [GitHub repository](https://github.com/GeWu-Lab/ROMA) for details.
|
| 105 |
|