README.md

---
title: OmniAgent-RL
emoji: 🎬
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
short_description: Agentic video understanding with active perception loop
python_version: "3.12"
startup_duration_timeout: 1h
---

# OmniAgent-RL: Native Active Perception as Reasoning for Omni-Modal Understanding

This Space demonstrates **OmniAgent-RL-7B**, a model that performs agentic video understanding
using an **Observation-Thought-Action (OTA) loop**. Instead of consuming all frames at once,
the model iteratively decides which parts of a video to examine — requesting specific frames,
audio clips, or video segments — before producing a final answer.

## How it works

1. The model receives a video and a question.
2. At each step, it outputs a JSON object with `observation`, `think`, `confidence`, and `action`.
3. Valid actions: `get_frames`, `get_audio`, `get_clip`, or `answer`.
4. The environment executes the action (via ffmpeg), appends the media to the conversation,
   and the model continues to the next step.
5. The loop terminates when the model outputs `answer` or the step limit is reached.

## Model

- **Base**: Qwen2.5-Omni-7B
- **Checkpoint**: [harryhsing/OmniAgent-RL-7B](https://huggingface.co/harryhsing/OmniAgent-RL-7B)
- **Paper**: [2606.19341](https://huggingface.co/papers/2606.19341)
- **Code**: [GitHub](https://github.com/harryhsing/OmniAgent)