Papers
arxiv:2611.21553

BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents

Published on Nov 16
· Submitted by
Kanchana Ranasinghe
on Nov 19
Authors:
,
,
,
,
,
,
,

Abstract

BLIP-Live unifies data, metrics, inference, and training for proactive streaming video understanding, using the PAUC-F metric and achieving state-of-the-art results on proactive streaming benchmarks.

Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.

Community

Paper submitter

We present BLIP-Live, a framework for proactive streaming video understanding.

We release,

  1. BLIP-Live-780K Dataset: A high-quality proactive streaming dataset for training built through
    automated generation of proactive question–response pairs, followed by comprehensive human
    review and filtering to ensure accurate response content and timing annotations.
  2. PAUC-F Metric: Evaluation metric designed to measure proactive understanding in VLMs that
    captures both response timeliness and firing precision.
  3. Inference Harness: Model-agnostic harness tuning and calibration that adapts offline VLMs
    for proactive responding under streaming style inference.
  4. Training Framework: Efficient training pipeline resulting in BLIP-Live-Gemma and BLIP-Live-Qwen models that yield relative improvements up to 44% over prior baselines across benchmarks.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.21553
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.21553 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.21553 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.21553 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.