Upload folder using huggingface_hub

README.md

@@ -47,25 +47,33 @@ Run every command **from the repo root** — modules use repo-root-relative `dat
 ```bash
 python -m venv .venv && .venv\Scripts\activate        # Windows (use: source .venv/bin/activate on *nix)
 pip install -e .                                       # installs humomni.* + deps (pyproject.toml)
-cp .env.example .env                                   # (cmd.exe: copy) then fill in the keys (below)
+cp .env.example .env                                   # then fill in the keys (below)
 ```
 `.env` needs `OPENROUTER_API_KEY` (the VLM) and `VERTEX_KEY` (the Gemini judge used for the LLM-entailment
-dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set
-(the official Track-2 phase-1 release) at `data/phase1/data/<id>/` — frames `0.5.jpg, 1.0.jpg, …` plus
-`question.json` per sample. (This is the `data/<id>/` layout the report appendix refers to, rooted at
-`data/phase1/data/`; a different location needs `--data_dir`, and a wrong path now aborts loudly.)
+dedup) — only for regenerating decisions from frames; the offline replay needs no keys. Put the test set at
+`data/phase1/data/<id>/` — frames `0.5.jpg, 1.0.jpg, …` (`.png` also supported, as shipped with the
+phase-2 test set) plus `question.json` per sample; `--data_dir` selects the test set (per-prompt caches
+are kept separate per phase automatically).
 
 ## Build the submission — one command
 `config.json` holds the entire method (`"ensemble": ["v5","v3","v8"] @ 40`, `dedup`/`theta`, and the
 perception `"gate"` disabled by default — `use_gate:false` gives the shipped max-score point; enabling it
 at novelty ≥ 16 / silence 3 s selects a lower-cost operating point):
 ```bash
-python scripts/build_submission.py --out submissions/tryanderror2.jsonl
+# phase-1 test set
+python scripts/build_submission.py --data_dir data/phase1/data --out submissions/tryanderror2.jsonl
+# phase-2 test set — same pipeline; its predictions use the EXH+COMP @ k32 operating point
+# (config_phase2.json), selected for cost; see Table I of the report
+python scripts/build_submission.py --data_dir data/phase2/data --config config_phase2.json --out submissions/tryanderror2_phase2.jsonl
 ```
-For each ensemble prompt it ensures a per-prompt VLM decision cache — **building it from the test frames
-if missing** (paid: ~1 VLM call per prompt per frame; resumable, skips cached frames) — then replays the
-merge + LLM-entailment dedup and **validates** the output (all 500 ids present, times multiples of 0.5).
-Re-runs are free once the caches exist.
+`--data_dir` points at any folder of `<id>/` sample dirs (frames `0.5.jpg`/`0.5.png`, `1.0.…`, plus
+`question.json`; both extensions are auto-detected). For each ensemble prompt it ensures a per-prompt VLM
+decision cache — **building it from the test frames if missing** (paid: ~1 VLM call per prompt per frame;
+resumable, skips cached frames) — then replays the merge + LLM-entailment dedup and **validates** the
+output (all ids present, times multiples of 0.5). Re-runs are free once the caches exist. Runs on
+different test sets never mix: decision/novelty caches are namespaced per data folder
+(`cache/phase1_*.jsonl` vs `cache/phase2_*.jsonl`), and the resume file is fingerprinted by
+(data path + config) so a leftover from a different run is set aside instead of resumed.
 
 The live equivalent produces byte-identical output (paid on every run — use when caches are absent):
 ```bash
@@ -80,3 +88,13 @@ Both drive the same `APIPolicy` from `config.json`, so all three commands point
 python tests/test_causality.py          # causality assert fires on shuffled / out-of-order frames
 python tests/test_inference_parity.py   # run_inference == infer_simple; ensemble multi-emit per frame
 ```
+
+## Compliance
+Track 2 permits any open-source or commercial LLM API (organizer clarification, 2026-07-07), with one
+condition: **when generating a response at time t, only frames at time ≤ t may be used as visual input.**
+This system enforces that structurally (below) and tests it.
+
+## The one hard rule — causality
+Acting at time *t* uses **only** frames with timestamp ≤ *t*. Enforced structurally in
+`humomni.core.streaming_driver.drive` (`assert t > last_t`) and guarded by `tests/test_causality.py`
+(the shuffled-frame test). Keep it green always.

config_phase2.json

@@ -0,0 +1,22 @@
+{
+  "theta": 0.6,
+  "gate": {
+    "use_gate": false,
+    "novelty": 16,
+    "max_silence_s": 3.0,
+    "warmup_s": 1.0
+  },
+  "dedup": {
+    "sim_high": 0.97,
+    "sim_low": 0.55,
+    "recent_k": 16,
+    "use_llm": true
+  },
+  "ensemble": {
+    "prompts": [
+      "v5",
+      "v3"
+    ],
+    "recent_k": 32
+  }
+}

paper_experiments/README.md

@@ -7,7 +7,7 @@ calls** and needs **no keys**.
 ## Setup (once)
 
 ```bash
-pip install -e .        # from the repo root; the run itself is fully offline (embeddings ship as a cache)
+pip install -e .        # from the repo root; first run downloads the MiniLM embedder (~90 MB)
 ```
 
 ## Run
@@ -32,7 +32,6 @@ Runs in a few minutes on CPU.
 | `data/gold_holdout.json` | the 100 untouched holdout items (public WEB split, items 400–499) |
 | `caches/web_{base,comp,exh,fs,fs_alt}.jsonl` | per-frame VLM decisions for each prompt (temperature 0), holdout videos only |
 | `caches/judge_cache.json` | every LLM-judge / entailment result, keyed by input hash |
-| `caches/emb_cache.{npy,keys.json}` | MiniLM embeddings of every cached draft (the cosine-dedup prefilter), so the replay never loads — or downloads — the embedder |
 
 `run.py` replays the cached decisions through the *same* merge + entailment-dedup code the live system
 runs (`humomni.phase1.dedup.EntailmentGuard`), then scores with the official-protocol harness

paper_experiments/run.py

@@ -13,8 +13,7 @@
 # with the official-protocol harness using the shipped LLM-judge cache (caches/judge_cache.json) —
 # so a rerun makes ZERO API calls and reproduces the reported numbers exactly. The perception gate is
 # replayed from the shipped per-frame novelty cache (caches/novelty_holdout.json). Requires only
-# `pip install -e .`; the draft embeddings ship too (caches/emb_cache.*), so no model download —
-# the run is fully offline.
+# `pip install -e .` (first run downloads the MiniLM sentence embedder, ~90 MB).
 import argparse
 import json
 import os
@@ -50,9 +49,6 @@ for c in CACHES.values():
 _TIMES = {q: sorted(ts) for q, ts in _TIMES.items()}
 
 emb_cache.load_disk()
-emb_cache.load_disk(os.path.join(HERE, "caches", "emb_cache"))   # shipped draft embeddings —
-# covers every draft below, so the MiniLM model (a ~90 MB first-run download) is never needed
-# for the replay; embed_many only encodes cache MISSES.
 emb_cache.embed_many([d["draft"].strip() for c in CACHES.values()
                       for d in c.mem.values() if d.get("draft", "").strip()])
 

pyproject.toml

@@ -7,10 +7,8 @@ name = "humomni-track2"
 version = "0.1.0"
 description = "HumOmni 2026 Track 2 (ProactivEval) — causal proactive streaming video QA (Phase-1 API orchestrator)"
 requires-python = ">=3.10"
-# Pinned to the versions the reported results were produced with (report Appendix, "Libraries").
 dependencies = [
-  "openai==2.43.0", "pillow==12.2.0", "numpy==2.2.6",
-  "scikit-learn==1.7.2", "sentence-transformers==5.6.0", "tqdm==4.68.3",
+  "openai", "pillow", "numpy", "scikit-learn", "sentence-transformers", "tqdm",
 ]
 # Note: the Gemini judge talks to Vertex AI express mode over plain REST (stdlib urllib) — no SDK dep.
 

scripts/build_submission.py

@@ -42,13 +42,15 @@ def _checkpoint(every=20):
             judges.jc_save()
 
 
+def _phase_of(data_dir):
+    """Phase tag derived from the data path: data/phase1/data -> 'phase1', data/phase2/data -> 'phase2'.
+    Keeps the per-prompt decision caches and the gate-novelty cache of different test sets separate,
+    so the SAME command works on any phase by changing only --data_dir."""
+    parent = os.path.basename(os.path.dirname(os.path.normpath(data_dir)))
+    return parent or os.path.basename(os.path.normpath(data_dir)) or "phase1"
+
+
 def build(data_dir, out, cfg, cache_tpl, workers, limit=0, build_workers=8):
-    # Fail LOUDLY on a wrong/empty data_dir: an empty gold set would otherwise sail through
-    # validation ("wrote 0 samples ... VALID") and produce an uploadable-looking empty file.
-    if not any(os.path.isdir(x) for x in glob.glob(os.path.join(data_dir, "*"))):
-        sys.exit(f"no sample dirs found under {data_dir!r} — expected the test set at "
-                 f"data/phase1/data/<id>/ (frames 0.5.jpg, 1.0.jpg, ... + question.json), "
-                 f"or pass --data_dir")
     ens = cfg.get("ensemble")
     if not ens:
         sys.exit("config.json has no 'ensemble' block — the best method is the v5+v3 ensemble; add it.")
@@ -64,7 +66,7 @@ def build(data_dir, out, cfg, cache_tpl, workers, limit=0, build_workers=8):
     novelty = None
     if g.get("use_gate"):
         from humomni.tuning.cache import build_novelty
-        nov_path = "cache/phase1_novelty.json"
+        nov_path = f"cache/{_phase_of(data_dir)}_novelty.json"
         if not os.path.exists(nov_path):
             print(f"[gate] computing per-frame novelty for {data_dir} -> {nov_path} (CPU, one-off) …", flush=True)
             build_novelty(data_dir=data_dir, out=nov_path)   # ALWAYS full — a --limit-partial cache would poison later runs
@@ -96,17 +98,38 @@ def build(data_dir, out, cfg, cache_tpl, workers, limit=0, build_workers=8):
     # Sample-level RESUME: completed records are appended to a `<out>.partial` sidecar as they finish,
     # so a killed build resumes instead of restarting (the dedup judge cache is already persisted
     # incrementally). The final `out` is written — and the sidecar dropped — only once all ids are done.
+    # The sidecar's first line is a FINGERPRINT of (data_dir + method config): a leftover partial from a
+    # different test set or a different config is set aside instead of resumed, so two runs can never mix
+    # samples computed under different settings into one output.
+    import hashlib
+    fp = hashlib.md5((os.path.normpath(data_dir) + "\x00"
+                      + json.dumps(cfg, sort_keys=True, ensure_ascii=False)).encode("utf-8")).hexdigest()
     partial = out + ".partial"
     done = {}
     if os.path.exists(partial):
-        for line in open(partial, encoding="utf-8"):
+        lines = open(partial, encoding="utf-8").read().splitlines()
+        meta_ok = False
+        if lines:
             try:
-                done[json.loads(line)["question_id"]] = line.rstrip("\n")
+                meta_ok = json.loads(lines[0]).get("__partial_config__") == fp
             except Exception:
-                pass
+                meta_ok = False
+        if meta_ok:
+            for line in lines[1:]:
+                try:
+                    done[json.loads(line)["question_id"]] = line
+                except Exception:
+                    pass
+        else:
+            os.replace(partial, partial + ".stale")
+            print(f"[resume] {partial} was built under a DIFFERENT data set or config — "
+                  f"set aside as .stale, starting fresh", flush=True)
     if done:
         print(f"[resume] {len(done)} samples already done in {partial}", flush=True)
     pw = open(partial, "a", encoding="utf-8")
+    if not done and os.path.getsize(partial) == 0:     # fresh sidecar -> stamp the fingerprint header
+        pw.write(json.dumps({"__partial_config__": fp}) + "\n")
+        pw.flush()
     pwlock = threading.Lock()
 
     def one(sd):
@@ -159,13 +182,17 @@ if __name__ == "__main__":
     ap.add_argument("--data_dir", default="data/phase1/data")
     ap.add_argument("--out", default="submissions/submission.jsonl")
     ap.add_argument("--config", default="config.json")
-    ap.add_argument("--cache_tpl", default="cache/phase1_{}.jsonl", help="per-prompt cache path ({} = prompt name)")
+    ap.add_argument("--cache_tpl", default=None,
+                    help="per-prompt cache path ({} = prompt name); default derives from --data_dir, "
+                         "e.g. data/phase2/data -> cache/phase2_{}.jsonl")
     ap.add_argument("--workers", type=int, default=1,
                     help="parallel samples. Keep at 1 on a COLD judge cache (the LLM dedup floods Vertex → 429); "
                          "raise to 3-4 once cached (re-run is then free).")
     ap.add_argument("--limit", type=int, default=0, help="only the first N samples (quick correctness check)")
     ap.add_argument("--build_workers", type=int, default=8, help="parallelism for BUILDING missing per-prompt VLM caches")
     a = ap.parse_args()
+    if not a.cache_tpl:
+        a.cache_tpl = f"cache/{_phase_of(a.data_dir)}_{{}}.jsonl"
     cfg = json.load(open(a.config, encoding="utf-8"))
     os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True)
 

scripts/infer_simple.py

@@ -25,13 +25,14 @@ def make_policy(name):
 
 def _video_length(sample_dir):
     """The latest frame timestamp present — the 'video_length' the loop steps up to (frames sit on a
-    0.5 s grid: 0.5.jpg, 1.0.jpg, …)."""
+    0.5 s grid: 0.5.jpg / 0.5.png, 1.0.jpg / 1.0.png, …)."""
     ts = []
-    for p in glob.glob(os.path.join(sample_dir, "*.jpg")):
-        try:
-            ts.append(float(os.path.basename(p)[:-4]))
-        except ValueError:
-            pass                                             # skip non-frame files
+    for pat in ("*.jpg", "*.png"):                           # phase-1 ships .jpg, phase-2 .png
+        for p in glob.glob(os.path.join(sample_dir, pat)):
+            try:
+                ts.append(float(os.path.basename(p)[:-4]))
+            except ValueError:
+                pass                                         # skip non-frame files
     return max(ts) if ts else 0.0
 
 
@@ -60,6 +61,8 @@ def infer_one(sample_dir, policy):
         for step in range(1, int(round(video_length / 0.5)) + 1):       # current_time = 0.5 .. video_length
             current_time = step * 0.5
             frame_path = os.path.join(sample_dir, f"{current_time:.1f}.jpg")
+            if not os.path.exists(frame_path):                          # phase-2 frames are .png
+                frame_path = os.path.join(sample_dir, f"{current_time:.1f}.png")
             if not os.path.exists(frame_path):                          # missing tick on the grid — skip
                 continue
             assert current_time > last_t, "frames must arrive in order (no future frames)"

src/humomni/core/streaming_driver.py

@@ -16,12 +16,13 @@ def iter_frames(sample_dir):
     Non-frame files (e.g. question.json) and any unparsable stems are skipped.
     """
     items = []
-    for f in glob.glob(os.path.join(sample_dir, "*.jpg")):
-        stem = os.path.basename(f)[:-4]
-        try:
-            items.append((float(stem), f))
-        except ValueError:
-            pass  # skip non-frame files
+    for pat in ("*.jpg", "*.png"):     # phase-1 test data ships .jpg frames, phase-2 ships .png
+        for f in glob.glob(os.path.join(sample_dir, pat)):
+            stem = os.path.basename(f)[:-4]
+            try:
+                items.append((float(stem), f))
+            except ValueError:
+                pass  # skip non-frame files
     items.sort(key=lambda x: x[0])
     out, last = [], None
     for t, f in items:                 # drop a malformed duplicate-timestamp frame (would trip drive()'s assert)

src/humomni/core/validate_submission.py

@@ -10,10 +10,6 @@ import sys
 
 def validate(path, gold_ids):
     seen, errs = set(), []
-    if not gold_ids:
-        # An empty gold set would validate ANY file (even an empty one) vacuously — that is
-        # always a mispathed --data_dir, never a real check.
-        return ["gold id list is empty — check --data_dir (expected data/phase1/data/<id>/)"]
     with open(path, encoding="utf-8") as f:
         for i, ln in enumerate(f, 1):
             ln = ln.strip()

src/humomni/tuning/cache.py

@@ -53,12 +53,20 @@ class VLMCache:
         return len(self.mem)
 
 
+def _frames_in(d):
+    """Ascending (t, path) for every frame image in dir d — stems on the 0.5-s grid, e.g. 12.5.jpg.
+    Both extensions are supported: phase-1 test data ships .jpg frames, phase-2 ships .png."""
+    out = []
+    for pat in ("*.jpg", "*.png"):
+        for p in glob.glob(os.path.join(d, pat)):
+            out.append((float(os.path.basename(p)[:-4]), p))
+    return sorted(out, key=lambda x: x[0])
+
+
 def frames_of(qid, data_dir="data/web_frames"):
     """Ascending (t, path) for a question_id (dir name drops the .mp4 suffix)."""
     name = qid[:-4] if qid.endswith(".mp4") else qid
-    d = os.path.join(data_dir, name)
-    return sorted([(float(os.path.basename(p)[:-4]), p) for p in glob.glob(d + "/*.jpg")],
-                  key=lambda x: x[0])
+    return _frames_in(os.path.join(data_dir, name))
 
 
 def _sample_dirs(data_dir, limit):
@@ -77,8 +85,7 @@ def build_web_cache(data_dir="data/web_frames", cache_path=CACHE_PATH, limit=0,
     for d in _sample_dirs(data_dir, limit):
         q = json.load(open(os.path.join(d, "question.json"), encoding="utf-8"))
         qid, question = q["question_id"], q["question"]
-        fr = sorted([(float(os.path.basename(p)[:-4]), p) for p in glob.glob(d + "/*.jpg")],
-                    key=lambda x: x[0])
+        fr = _frames_in(d)
         for i, (t, _) in enumerate(fr):
             if cache.get(qid, t) is None:
                 tasks.append((qid, question, fr[:i + 1], t))
@@ -109,8 +116,7 @@ def build_novelty(data_dir="data/web_frames", out=NOVELTY_PATH, limit=0):
     res = {}
     for d in _sample_dirs(data_dir, limit):
         qid = json.load(open(os.path.join(d, "question.json"), encoding="utf-8"))["question_id"]
-        fr = sorted([(float(os.path.basename(p)[:-4]), p) for p in glob.glob(d + "/*.jpg")],
-                    key=lambda x: x[0])
+        fr = _frames_in(d)
         prev, nov = None, {}
         for t, p in fr:
             f = np.asarray(Image.open(p).convert("L").resize((64, 36)), dtype=np.float32)