HumOmni 2026 · Track 2 (ProactivEval) — proactive streaming video QA

Watch a video at 2 fps, given one question at t=0, and decide when to speak and what to say — causally (using only frames with timestamp ≤ t). Ranking score = 0.9·PAUC + 0.1·(1−duplicate); hidden judge = Gemini 3.1 Flash-Lite. Baseline MMDuet2 = 53.3; public leader at the time = 60.21.

This repository is the Phase-1 system: a causal streaming orchestrator around a hosted VLM (qwen/qwen3.7-plus via OpenRouter). No training and no local large models — it runs on a single 8 GB GPU.

The method in one paragraph

The hidden judge is recall-only — it asks whether the accumulated response text covers the reference answer's key points, with no precision penalty for extra detail. So the winning strategy is comprehensive coverage: fire on every frame (gate off), prompt the VLM to describe the scene exhaustively and read all on-screen text verbatim, and strip only true repeats with an LLM-entailment dedup. The final config queries a small prompt ensemble per frame — v5 (exhaustive + OCR) + v3 (comprehensive) + v8 (few-shot, teach-by-example) — and merges their drafts. The big lever was the prompt, not timing or a bigger model.

Result (WEB dev set, faithful Gemini judge, untouched holdout gold[400:500])

stage	score	note
baseline (focused prompt + throttle gate)	~0.61	the original leaderboard submission
+ gate-off (fire every frame)	0.625	throttling loses coverage
+ comprehensive prompt (v3)	0.733	recall-only judge ⇒ coverage wins
+ exhaustive + OCR (v5), recent_k=16	0.7761	best single prompt
+ v5+v3 ensemble @ recent_k=32	0.7909	robust 2× config
+ v8 few-shot (v5+v3+v8 @ k40)	0.8018	the final submission (gate off)
+ perception gate (novelty ≥ 16, silence 3 s)	0.7853	optional lower-cost operating point (−51% calls)

Caveat: WEB-dev is the leakage-free dev set, not the phase1 test set (which has no ground truth). The v8 gain is partly holdout-overfit — the robust gain over v5+v3 is ~+0.002–0.005. Our PAUC harness is verified bit-exact against the official scorer, and the grader is confirmed Gemini 3.1 Flash-Lite.

Project layout

src/humomni/
  core/     streaming_driver · context · pauc_eval · judges · emb_cache · env_util · validate_submission
  phase1/   vlm_client · dedup · policy_api · policies · perception_gate
  tuning/   cache · faithful_eval
scripts/    build_submission.py · run_inference.py · infer_simple.py
tests/      test_causality.py · test_inference_parity.py
config.json   .env.example   pyproject.toml

Run every command from the repo root — modules use repo-root-relative data/ and cache/ paths.

Setup

python -m venv .venv && .venv\Scripts\activate        # Windows (use: source .venv/bin/activate on *nix)
pip install -e .                                       # installs humomni.* + deps (pyproject.toml)
cp .env.example .env                                   # (cmd.exe: copy) then fill in the keys (below)

.env needs OPENROUTER_API_KEY (the VLM) and VERTEX_KEY (the Gemini judge used for the LLM-entailment dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set (the official Track-2 phase-1 release) at data/phase1/data/<id>/ — frames 0.5.jpg, 1.0.jpg, … plus question.json per sample. (This is the data/<id>/ layout the report appendix refers to, rooted at data/phase1/data/; a different location needs --data_dir, and a wrong path now aborts loudly.)

Build the submission — one command

config.json holds the entire method ("ensemble": ["v5","v3","v8"] @ 40, dedup/theta, and the perception "gate" disabled by default — use_gate:false gives the shipped max-score point; enabling it at novelty ≥ 16 / silence 3 s selects a lower-cost operating point):

python scripts/build_submission.py --out submissions/tryanderror2.jsonl

For each ensemble prompt it ensures a per-prompt VLM decision cache — building it from the test frames if missing (paid: ~1 VLM call per prompt per frame; resumable, skips cached frames) — then replays the merge + LLM-entailment dedup and validates the output (all 500 ids present, times multiples of 0.5). Re-runs are free once the caches exist.

The live equivalent produces byte-identical output (paid on every run — use when caches are absent):

python scripts/run_inference.py --policy api --workers 8 --out submissions/tryanderror2.jsonl
python scripts/infer_simple.py  --policy api             --out submissions/tryanderror2.jsonl   # minimal twin

Both drive the same APIPolicy from config.json, so all three commands point at the same method (live ≡ replay). Upload is manual: the file must be named exactly your Team ID.

Tests

python tests/test_causality.py          # causality assert fires on shuffled / out-of-order frames
python tests/test_inference_parity.py   # run_inference == infer_simple; ensemble multi-emit per frame