| # Paper experiments — one command per table |
|
|
| Reproduces every number in the report's tables, **deterministically and offline**: the per-frame VLM |
| decisions (holdout videos) and all LLM-judge results are shipped as caches, so a rerun makes **zero API |
| calls** and needs **no keys**. |
|
|
| ## Setup (once) |
|
|
| ```bash |
| pip install -e . # from the repo root; the run itself is fully offline (embeddings ship as a cache) |
| ``` |
|
|
| ## Run |
|
|
| ```bash |
| python paper_experiments/run.py table1 # Table 1 — main progression (score AND cost) |
| python paper_experiments/run.py table2 # Table 2 — ensemble ablation (calls per frame) |
| python paper_experiments/run.py table3 # Table 3 — dedup ablation |
| python paper_experiments/run.py gate # Table 4 — perception-gate cost/score frontier |
| python paper_experiments/run.py sensitivity # few-shot example-sensitivity check |
| python paper_experiments/run.py all |
| ``` |
|
|
| Each line prints `score` (0.9·PAUC + 0.1·(1−dup)), `PAUC`, `dup`, mean emissions per video, and the |
| cost: mean **VLM calls per video** (+ an approximate US$ figure at our measured ≈$25 / 17.3k calls). |
| Runs in a few minutes on CPU. |
|
|
| ## What is shipped, and why the numbers are exact |
|
|
| | file | content | |
| |---|---| |
| | `data/gold_holdout.json` | the 100 untouched holdout items (public WEB split, items 400–499) | |
| | `caches/web_{base,comp,exh,fs,fs_alt}.jsonl` | per-frame VLM decisions for each prompt (temperature 0), holdout videos only | |
| | `caches/judge_cache.json` | every LLM-judge / entailment result, keyed by input hash | |
| | `caches/emb_cache.{npy,keys.json}` | MiniLM embeddings of every cached draft (the cosine-dedup prefilter), so the replay never loads — or downloads — the embedder | |
|
|
| `run.py` replays the cached decisions through the *same* merge + entailment-dedup code the live system |
| runs (`humomni.phase1.dedup.EntailmentGuard`), then scores with the official-protocol harness |
| (`humomni.tuning.faithful_eval`). Since the VLM decisions, judge results, and dedup logic are all |
| deterministic replays, the printed numbers match the report exactly. |
|
|
| To regenerate the VLM caches from scratch instead (paid; OpenRouter + Vertex keys in `.env`): |
| `python -m humomni.tuning.cache --func vlm` with the corresponding prompt — see the repo README. |
|
|
| Prompt naming: EXH / COMP / FS in the report = `v5` / `v3` / `v8` in the code; `fs_alt` = `v10` |
| (the example-swapped variant used in the sensitivity check). |
|
|
