| # HumOmni 2026 · Track 2 (ProactivEval) — proactive streaming video QA | |
| Watch a video at 2 fps, given one question at t=0, and decide **when** to speak and **what** to | |
| say — causally (using only frames with timestamp ≤ t). Ranking score = **0.9·PAUC + 0.1·(1−duplicate)**; | |
| hidden judge = Gemini 3.1 Flash-Lite. Baseline MMDuet2 = 53.3; public leader at the time = 60.21. | |
| This repository is the **Phase-1 system**: a causal streaming orchestrator around a hosted VLM | |
| (`qwen/qwen3.7-plus` via OpenRouter). No training and no local large models — it runs on a single 8 GB GPU. | |
| ## The method in one paragraph | |
| The hidden judge is **recall-only** — it asks whether the accumulated response text *covers* the | |
| reference answer's key points, with **no precision penalty** for extra detail. So the winning strategy is | |
| **comprehensive coverage**: fire on every frame (gate off), prompt the VLM to describe the scene | |
| exhaustively and read all on-screen text verbatim, and strip only true repeats with an LLM-entailment | |
| dedup. The final config queries a small **prompt ensemble** per frame — `v5` (exhaustive + OCR) + `v3` | |
| (comprehensive) + `v8` (few-shot, teach-by-example) — and merges their drafts. The big lever was the | |
| **prompt**, not timing or a bigger model. | |
| ## Result (WEB dev set, faithful Gemini judge, untouched holdout `gold[400:500]`) | |
| | stage | score | note | | |
| |---|---|---| | |
| | baseline (focused prompt + throttle gate) | ~0.61 | the original leaderboard submission | | |
| | + gate-off (fire every frame) | 0.625 | throttling loses coverage | | |
| | + comprehensive prompt (`v3`) | 0.733 | recall-only judge ⇒ coverage wins | | |
| | + exhaustive + OCR (`v5`), recent_k=16 | 0.7761 | best single prompt | | |
| | + `v5`+`v3` ensemble @ recent_k=32 | 0.7909 | robust 2× config | | |
| | + `v8` few-shot (`v5`+`v3`+`v8` @ k40) | **0.8018** | **the final submission** (gate off) | | |
| | + perception gate (novelty ≥ 16, silence 3 s) | 0.7853 | optional lower-cost operating point (−51% calls) | | |
| **Caveat:** WEB-dev is the leakage-free dev set, **not** the phase1 test set (which has no ground truth). | |
| The `v8` gain is partly holdout-overfit — the robust gain over `v5`+`v3` is ~+0.002–0.005. Our PAUC harness | |
| is verified bit-exact against the official scorer, and the grader is confirmed Gemini 3.1 Flash-Lite. | |
| ## Project layout | |
| ``` | |
| src/humomni/ | |
| core/ streaming_driver · context · pauc_eval · judges · emb_cache · env_util · validate_submission | |
| phase1/ vlm_client · dedup · policy_api · policies · perception_gate | |
| tuning/ cache · faithful_eval | |
| scripts/ build_submission.py · run_inference.py · infer_simple.py | |
| tests/ test_causality.py · test_inference_parity.py | |
| config.json .env.example pyproject.toml | |
| ``` | |
| Run every command **from the repo root** — modules use repo-root-relative `data/` and `cache/` paths. | |
| ## Setup | |
| ```bash | |
| python -m venv .venv && .venv\Scripts\activate # Windows (use: source .venv/bin/activate on *nix) | |
| pip install -e . # installs humomni.* + deps (pyproject.toml) | |
| cp .env.example .env # (cmd.exe: copy) then fill in the keys (below) | |
| ``` | |
| `.env` needs `OPENROUTER_API_KEY` (the VLM) and `VERTEX_KEY` (the Gemini judge used for the LLM-entailment | |
| dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set | |
| (the official Track-2 phase-1 release) at `data/phase1/data/<id>/` — frames `0.5.jpg, 1.0.jpg, …` plus | |
| `question.json` per sample. (This is the `data/<id>/` layout the report appendix refers to, rooted at | |
| `data/phase1/data/`; a different location needs `--data_dir`, and a wrong path now aborts loudly.) | |
| ## Build the submission — one command | |
| `config.json` holds the entire method (`"ensemble": ["v5","v3","v8"] @ 40`, `dedup`/`theta`, and the | |
| perception `"gate"` disabled by default — `use_gate:false` gives the shipped max-score point; enabling it | |
| at novelty ≥ 16 / silence 3 s selects a lower-cost operating point): | |
| ```bash | |
| python scripts/build_submission.py --out submissions/tryanderror2.jsonl | |
| ``` | |
| For each ensemble prompt it ensures a per-prompt VLM decision cache — **building it from the test frames | |
| if missing** (paid: ~1 VLM call per prompt per frame; resumable, skips cached frames) — then replays the | |
| merge + LLM-entailment dedup and **validates** the output (all 500 ids present, times multiples of 0.5). | |
| Re-runs are free once the caches exist. | |
| The live equivalent produces byte-identical output (paid on every run — use when caches are absent): | |
| ```bash | |
| python scripts/run_inference.py --policy api --workers 8 --out submissions/tryanderror2.jsonl | |
| python scripts/infer_simple.py --policy api --out submissions/tryanderror2.jsonl # minimal twin | |
| ``` | |
| Both drive the same `APIPolicy` from `config.json`, so all three commands point at the same method | |
| (live ≡ replay). Upload is manual: the file must be named exactly your Team ID. | |
| ## Tests | |
| ```bash | |
| python tests/test_causality.py # causality assert fires on shuffled / out-of-order frames | |
| python tests/test_inference_parity.py # run_inference == infer_simple; ensemble multi-emit per frame | |
| ``` | |