# Paper experiments — one command per table

Reproduces every number in the report's tables, **deterministically and offline**: the per-frame VLM
decisions (holdout videos) and all LLM-judge results are shipped as caches, so a rerun makes **zero API
calls** and needs **no keys**.

## Setup (once)

```bash
pip install -e .        # from the repo root; the run itself is fully offline (embeddings ship as a cache)
```

## Run

```bash
python paper_experiments/run.py table1        # Table 1 — main progression (score AND cost)
python paper_experiments/run.py table2        # Table 2 — ensemble ablation (calls per frame)
python paper_experiments/run.py table3        # Table 3 — dedup ablation
python paper_experiments/run.py gate          # Table 4 — perception-gate cost/score frontier
python paper_experiments/run.py sensitivity   # few-shot example-sensitivity check
python paper_experiments/run.py all
```

Each line prints `score` (0.9·PAUC + 0.1·(1−dup)), `PAUC`, `dup`, mean emissions per video, and the
cost: mean **VLM calls per video** (+ an approximate US$ figure at our measured ≈$25 / 17.3k calls).
Runs in a few minutes on CPU.

## What is shipped, and why the numbers are exact

| file | content |
|---|---|
| `data/gold_holdout.json` | the 100 untouched holdout items (public WEB split, items 400–499) |
| `caches/web_{base,comp,exh,fs,fs_alt}.jsonl` | per-frame VLM decisions for each prompt (temperature 0), holdout videos only |
| `caches/judge_cache.json` | every LLM-judge / entailment result, keyed by input hash |
| `caches/emb_cache.{npy,keys.json}` | MiniLM embeddings of every cached draft (the cosine-dedup prefilter), so the replay never loads — or downloads — the embedder |

`run.py` replays the cached decisions through the *same* merge + entailment-dedup code the live system
runs (`humomni.phase1.dedup.EntailmentGuard`), then scores with the official-protocol harness
(`humomni.tuning.faithful_eval`). Since the VLM decisions, judge results, and dedup logic are all
deterministic replays, the printed numbers match the report exactly.

To regenerate the VLM caches from scratch instead (paid; OpenRouter + Vertex keys in `.env`):
`python -m humomni.tuning.cache --func vlm` with the corresponding prompt — see the repo README.

Prompt naming: EXH / COMP / FS in the report = `v5` / `v3` / `v8` in the code; `fs_alt` = `v10`
(the example-swapped variant used in the sensitivity check).