# HumOmni 2026 · Track 2 (ProactivEval) — proactive streaming video QA

Watch a video at 2 fps, given one question at t=0, and decide **when** to speak and **what** to
say — causally (using only frames with timestamp ≤ t). Ranking score = **0.9·PAUC + 0.1·(1−duplicate)**;
hidden judge = Gemini 3.1 Flash-Lite. Baseline MMDuet2 = 53.3; public leader at the time = 60.21.

This repository is the **Phase-1 system**: a causal streaming orchestrator around a hosted VLM
(`qwen/qwen3.7-plus` via OpenRouter). No training and no local large models — it runs on a single 8 GB GPU.

## The method in one paragraph
The hidden judge is **recall-only** — it asks whether the accumulated response text *covers* the
reference answer's key points, with **no precision penalty** for extra detail. So the winning strategy is
**comprehensive coverage**: fire on every frame (gate off), prompt the VLM to describe the scene
exhaustively and read all on-screen text verbatim, and strip only true repeats with an LLM-entailment
dedup. The final config queries a small **prompt ensemble** per frame — `v5` (exhaustive + OCR) + `v3`
(comprehensive) + `v8` (few-shot, teach-by-example) — and merges their drafts. The big lever was the
**prompt**, not timing or a bigger model.

## Result (WEB dev set, faithful Gemini judge, untouched holdout `gold[400:500]`)
| stage | score | note |
|---|---|---|
| baseline (focused prompt + throttle gate) | ~0.61 | the original leaderboard submission |
| + gate-off (fire every frame) | 0.625 | throttling loses coverage |
| + comprehensive prompt (`v3`) | 0.733 | recall-only judge ⇒ coverage wins |
| + exhaustive + OCR (`v5`), recent_k=16 | 0.7761 | best single prompt |
| + `v5`+`v3` ensemble @ recent_k=32 | 0.7909 | robust 2× config |
| + `v8` few-shot (`v5`+`v3`+`v8` @ k40) | **0.8018** | **the final submission** (gate off) |
| + perception gate (novelty ≥ 16, silence 3 s) | 0.7853 | optional lower-cost operating point (−51% calls) |

**Caveat:** WEB-dev is the leakage-free dev set, **not** the phase1 test set (which has no ground truth).
The `v8` gain is partly holdout-overfit — the robust gain over `v5`+`v3` is ~+0.002–0.005. Our PAUC harness
is verified bit-exact against the official scorer, and the grader is confirmed Gemini 3.1 Flash-Lite.

## Project layout
```
src/humomni/
  core/     streaming_driver · context · pauc_eval · judges · emb_cache · env_util · validate_submission
  phase1/   vlm_client · dedup · policy_api · policies · perception_gate
  tuning/   cache · faithful_eval
scripts/    build_submission.py · run_inference.py · infer_simple.py
tests/      test_causality.py · test_inference_parity.py
config.json   .env.example   pyproject.toml
```
Run every command **from the repo root** — modules use repo-root-relative `data/` and `cache/` paths.

## Setup
```bash
python -m venv .venv && .venv\Scripts\activate        # Windows (use: source .venv/bin/activate on *nix)
pip install -e .                                       # installs humomni.* + deps (pyproject.toml)
cp .env.example .env                                   # (cmd.exe: copy) then fill in the keys (below)
```
`.env` needs `OPENROUTER_API_KEY` (the VLM) and `VERTEX_KEY` (the Gemini judge used for the LLM-entailment
dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set
(the official Track-2 phase-1 release) at `data/phase1/data/<id>/` — frames `0.5.jpg, 1.0.jpg, …` plus
`question.json` per sample. (This is the `data/<id>/` layout the report appendix refers to, rooted at
`data/phase1/data/`; a different location needs `--data_dir`, and a wrong path now aborts loudly.)

## Build the submission — one command
`config.json` holds the entire method (`"ensemble": ["v5","v3","v8"] @ 40`, `dedup`/`theta`, and the
perception `"gate"` disabled by default — `use_gate:false` gives the shipped max-score point; enabling it
at novelty ≥ 16 / silence 3 s selects a lower-cost operating point):
```bash
python scripts/build_submission.py --out submissions/tryanderror2.jsonl
```
For each ensemble prompt it ensures a per-prompt VLM decision cache — **building it from the test frames
if missing** (paid: ~1 VLM call per prompt per frame; resumable, skips cached frames) — then replays the
merge + LLM-entailment dedup and **validates** the output (all 500 ids present, times multiples of 0.5).
Re-runs are free once the caches exist.

The live equivalent produces byte-identical output (paid on every run — use when caches are absent):
```bash
python scripts/run_inference.py --policy api --workers 8 --out submissions/tryanderror2.jsonl
python scripts/infer_simple.py  --policy api             --out submissions/tryanderror2.jsonl   # minimal twin
```
Both drive the same `APIPolicy` from `config.json`, so all three commands point at the same method
(live ≡ replay). Upload is manual: the file must be named exactly your Team ID.

## Tests
```bash
python tests/test_causality.py          # causality assert fires on shuffled / out-of-order frames
python tests/test_inference_parity.py   # run_inference == infer_simple; ensemble multi-emit per frame
```