Papers
arxiv:2607.11523

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Published on Jul 13
· Submitted by
Tim Kang
on Jul 16
Authors:
,
,
,
,
,
,
,
,

Abstract

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at https://sitonggong.github.io/EgoServe-page/{Vinci2}.

Community

Paper submitter

🎯 Vinci2: Proactive Assistance in Continuous Egocentric Videos (ECCV 2026)

When should an AI assistant speak up—without being asked?

Today's assistants are stuck at two extremes: some wait passively for a query, others fire off a response to every event they detect. Neither asks the harder question—is this the right moment to intervene at all?

We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to decide when and whether help is actually welcome.

📊 EgoServe: the first large-scale benchmark for proactive assistance in continuous egocentric video: 3,000+ service instances spanning 4 temporal memory horizons (from split-second safety alerts to long-term habit coaching) across 10 service categories.

🧠 EgoMemo: a training-free, memory-augmented agent that fuses three complementary memories: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives, into retrieval-augmented reasoning that decides, at every timestep, whether to act.

Vinci2 pushes on-device egocentric assistance from reactive to truly proactive.

🔗 Project: https://sitonggong.github.io/EgoServe-page/
💻 Code: https://github.com/SitongGong/EgoMemo

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2607.11523 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2607.11523 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.