Update README.md

README.md

@@ -1,3 +1,84 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+library_name: transformers
+tags:
+- multimodal
+- video-understanding
+- audio-understanding
+- streaming
+- real-time
+- omni-modal
+pipeline_tag: video-text-to-text
+---
+
+# ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
+
+<div align="center">
+  <img src="[INSERT LINK TO FIGURE 2 (ARCHITECTURE) HERE]" width="800"/>
+  <p>Figure: ROMA processes streaming inputs as aligned multimodal units, using a 'Speak Head' to decide when to respond.</p>
+</div>
+
+## Model Summary
+
+[cite_start]**ROMA** is a Real-time Omni-Multimodal Assistant designed for unified streaming audio-video understanding[cite: 9, 46]. [cite_start]Unlike traditional video LLMs that only answer after a query, ROMA integrates both **Reactive** (Question Answering) and **Proactive** (Event-Driven Alert, Real-Time Narration) capabilities within a single framework[cite: 58].
+
+[cite_start]ROMA introduces a "Speak Head" mechanism to decouple response timing from content generation, allowing it to autonomously decide *when* to speak based on the continuous audio-visual stream[cite: 11, 49].
+
+- **Paper:** [ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding](https://arxiv.org/abs/250x.xxxxx)
+- [cite_start]**Project Page:** [Link](https://eureka-maggie.github.io/ROMA_show/) [cite: 20]
+- **Repository:** [INSERT GITHUB LINK]
+- **Developed by:** Institute of Computing Technology, CAS; UCAS; [cite_start]Tsinghua University[cite: 2, 3, 4].
+
+## Key Capabilities
+
+[cite_start]ROMA excels in three main interaction modes[cite: 53]:
+
+1.  [cite_start]**Event-Driven Alert (Proactive):** Monitors the stream and notifies the user immediately when a specific event occurs (e.g., "Notify me when a bird pops out")[cite: 23, 210].
+2.  [cite_start]**Real-Time Narration (Proactive):** Continuously describes the evolving video and audio context without needing user prompts[cite: 25, 223].
+3.  [cite_start]**Reactive QA:** Answers questions based on the past context, handling synchronized audio and video inputs[cite: 27, 227].
+
+## Model Architecture
+
+[cite_start]ROMA processes continuous inputs as synchronized **Multimodal Units** (1-second intervals aligning dense audio with video frames)[cite: 10, 128].
+
+Key architectural innovations include:
+- [cite_start]**Chunked TMROPE:** Ensures proper temporal position encoding across streaming chunks[cite: 48, 134].
+- [cite_start]**Speak Head:** A lightweight module parallel to the LM head that predicts a binary probability to trigger a response, solving the issue of task conflict between listening and speaking[cite: 144, 145].
+
+## Performance
+
+[cite_start]ROMA achieves state-of-the-art performance on proactive benchmarks while remaining competitive on reactive settings[cite: 14].
+
+| Benchmark Type | Task | ROMA Performance | State-of-the-Art? |
+| :--- | :--- | :--- | :--- |
+| **Proactive** | Event-Driven Alert (QVHighlights) | [cite_start]**53.7 mAP** [cite: 241] | ✅ Yes |
+| **Proactive** | Real-Time Narration (YouCook2) | [cite_start]**35.21 F1** [cite: 251] | ✅ Yes |
+| **Reactive** | Omni-Source Understanding (StreamingBench) | [cite_start]**Top Rank** [cite: 255] | ✅ Yes |
+| **Reactive** | Full-Modality QA (Video-MME w/ Audio) | [cite_start]**33.30 Accuracy** [cite: 260] | ✅ Yes |
+
+## Quick Start
+
+```python
+# Note: This is a pseudo-code example. Please refer to the official GitHub repo for the exact inference loop.
+
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained("Your-HF-Username/ROMA")
+tokenizer = AutoTokenizer.from_pretrained("Your-HF-Username/ROMA")
+
+# Input: Streaming chunks of Video and Audio (1-second units)
+# The model uses a "Speak Head" to decide when to output text.
+
+stream = load_video_audio_stream("example_video.mp4")
+
+history_cache = None
+
+for multimodal_unit in stream:
+    # Process 1-second unit
+    response, history_cache = model.streaming_inference(
+        multimodal_unit, 
+        past_key_values=history_cache
+    )
+    
+    if response:
+        print(f"ROMA says: {response}")