Paper experiments — one command per table
Reproduces every number in the report's tables, deterministically and offline: the per-frame VLM decisions (holdout videos) and all LLM-judge results are shipped as caches, so a rerun makes zero API calls and needs no keys.
Setup (once)
pip install -e . # from the repo root; the run itself is fully offline (embeddings ship as a cache)
Run
python paper_experiments/run.py table1 # Table 1 — main progression (score AND cost)
python paper_experiments/run.py table2 # Table 2 — ensemble ablation (calls per frame)
python paper_experiments/run.py table3 # Table 3 — dedup ablation
python paper_experiments/run.py gate # Table 4 — perception-gate cost/score frontier
python paper_experiments/run.py sensitivity # few-shot example-sensitivity check
python paper_experiments/run.py all
Each line prints score (0.9·PAUC + 0.1·(1−dup)), PAUC, dup, mean emissions per video, and the
cost: mean VLM calls per video (+ an approximate US$ figure at our measured ≈$25 / 17.3k calls).
Runs in a few minutes on CPU.
What is shipped, and why the numbers are exact
| file | content |
|---|---|
data/gold_holdout.json |
the 100 untouched holdout items (public WEB split, items 400–499) |
caches/web_{base,comp,exh,fs,fs_alt}.jsonl |
per-frame VLM decisions for each prompt (temperature 0), holdout videos only |
caches/judge_cache.json |
every LLM-judge / entailment result, keyed by input hash |
caches/emb_cache.{npy,keys.json} |
MiniLM embeddings of every cached draft (the cosine-dedup prefilter), so the replay never loads — or downloads — the embedder |
run.py replays the cached decisions through the same merge + entailment-dedup code the live system
runs (humomni.phase1.dedup.EntailmentGuard), then scores with the official-protocol harness
(humomni.tuning.faithful_eval). Since the VLM decisions, judge results, and dedup logic are all
deterministic replays, the printed numbers match the report exactly.
To regenerate the VLM caches from scratch instead (paid; OpenRouter + Vertex keys in .env):
python -m humomni.tuning.cache --func vlm with the corresponding prompt — see the repo README.
Prompt naming: EXH / COMP / FS in the report = v5 / v3 / v8 in the code; fs_alt = v10
(the example-swapped variant used in the sensitivity check).