README.md

metadata

title: OmniAgent-RL
emoji: 🎬
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
short_description: Agentic video understanding with active perception loop
python_version: '3.12'
startup_duration_timeout: 1h

OmniAgent-RL: Native Active Perception as Reasoning for Omni-Modal Understanding

This Space demonstrates OmniAgent-RL-7B, a model that performs agentic video understanding using an Observation-Thought-Action (OTA) loop. Instead of consuming all frames at once, the model iteratively decides which parts of a video to examine — requesting specific frames, audio clips, or video segments — before producing a final answer.

How it works

1. The model receives a video and a question.
2. At each step, it outputs a JSON object with observation, think, confidence, and action.
3. Valid actions: get_frames, get_audio, get_clip, or answer.
4. The environment executes the action (via ffmpeg), appends the media to the conversation, and the model continues to the next step.
5. The loop terminates when the model outputs answer or the step limit is reached.

Model

• Base: Qwen2.5-Omni-7B
• Checkpoint: harryhsing/OmniAgent-RL-7B
• Paper: 2606.19341
• Code: GitHub