Papers
arxiv:2611.17048

Beyond Answering: Learning to Speak First and Act in Streaming Video

Published on Nov 15
· Submitted by
He Yuyang
on Nov 18
Authors:
,
,
,
,
,
,
,
,
,
,
,

Abstract

AURA 2.0 is a streaming video large language model that can initiate conversations, call tools, and perform interactive visual grounding, with long-horizon memory and a three-stage training scheme for improved streaming interaction.

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.

Community

Paper author Paper submitter

Hi Hugging Face community! 👋 We’re excited to share AURA 2.0 — Beyond Answering: Learning to Speak First and Act in Streaming Video.

What if a video AI could notice something worth discussing and start the conversation, rather than always waiting for a question?

AURA 2.0 is an open-source streaming video understanding model that brings together three capabilities:

  • Model-Initiated Conversation (MIC): Decide whether to speak, when to speak, and what to say based on the evolving video and dialogue context—including when to remain silent.
  • Stream-Aware Tool Calling (SATC): Select tools and construct arguments from the current visual context, then incorporate tool results into the conversation.
  • Interactive Visual Grounding (IVG): Connect dialogue to specific objects through user-provided visual cues and model-generated, frame-aligned bounding boxes.
    The model is trained on approximately 5,400 hours of video and 40 million frames, with three-stage training and long-horizon interaction memory. It achieves the highest reported aggregate scores across IPIBench, StreamArena, ProactiveVideoQA, and PhoStream among the models compared in our report.

Explore the demos and find code, model downloads, and deployment instructions on our project page.

We welcome feedback, reproductions, and discussion—especially on how proactive video assistants should balance useful intervention with knowing when to wait!

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.17048
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.17048 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.