Upload folder using huggingface_hub

README.md

@@ -17,6 +17,10 @@ Pick the most **empathetic** spoken response to a human utterance. The candidate
 with essentially the same words — they differ only in **vocal delivery** — so this is a **listening**
 problem, not a text problem: the answer is in the prosody, never the transcript.
 
+This repository is the complete submission: the pre-trained model, the self-contained inference and
+evaluation code, the full training pipeline, the reproduction scripts, and the technical report
+(`report/technical_report.pdf`).
+
 ## The method
 
 A **single open-weights model**, run **offline** with **no API** and **no dataset-specific handling**:
@@ -32,7 +36,7 @@ A **single open-weights model**, run **offline** with **no API** and **no datase
 
 Scored against our expert annotations with the official metric: **FINAL = 0.832** (tone 0.933, context 0.805),
 **+0.285** over the 0.547 baseline. The method, ablations, environment, and reproduction details are in
-the technical report (submitted separately, not part of this repository).
+`report/technical_report.pdf`.
 
 ## Reproduce the submission
 
@@ -46,16 +50,8 @@ python inference/evaluate.py     # official metric vs. inference/annotations.jso
 ```
 
 First run downloads WavLM-large (~1.2 GB) from the Hub; afterwards it runs fully offline (GPU ≈10–15 min for
-530 questions, or CPU).
-
-> **Windows / PowerShell:** `.venv\Scripts\activate` can be silently blocked by the default script
-> execution policy, after which `pip` installs into the wrong Python. Safest is to skip activation and
-> call the venv's interpreter directly — `.venv\Scripts\python.exe -m pip install -r inference\requirements.txt`,
-> then `.venv\Scripts\python.exe inference\run.py`. After activating, `pip -V` must show a path inside `.venv`.
->
-> **GPU:** install the CUDA torch build *before* the requirements (see the note in `requirements.txt`).
-> If a CPU-only torch is already installed, pip will skip the CUDA build unless you force it:
-> `pip install torch --index-url https://download.pytorch.org/whl/cu128 --force-reinstall`
+530 questions, or CPU). Full step-by-step instructions — the runtime environment, libraries, and every
+parameter — are in `report/technical_report.pdf` (Appendix A) and `inference/README.md`.
 
 To retrain the ranker from scratch (the full pipeline that produced `inference/ranker.pkl`):
 
@@ -75,8 +71,16 @@ scripts/produce_submission.py   # full pipeline: source data -> train ranker ->
 experiments/                    # one-command reproduction of every table/finding in the report
 empathyeval/                    # release parsing + the 16 kHz-mono audio loader + the official metric
 configs/phase1.yaml             # data paths, audio, cache
+submissions/                    # the deliverable, <team_id>.jsonl
+report/                         # technical report (PDF + LaTeX source + references.bib)
+research/                       # ARCHIVE: all experiment code kept for the technical report (reference-only)
 ```
 
 **For the competition, only `inference/` is needed** — it is fully self-contained (the pre-trained model
 plus a pure-inference script) and, delivered alone, regenerates the exact result.
-`scripts/produce_submission.py` is the full pipeline that trained the ranker saved as `inference/ranker.pkl`.
+`scripts/produce_submission.py` is the full pipeline that trained the ranker saved as `inference/ranker.pkl`.
+
+## Compliance
+
+Inference and the submission are produced entirely from **open weights, offline, with no API and no dataset
+labels**. Any API-calling code lives under `research/` and was development-only — never in the submission path.

experiments/README.md

@@ -1,6 +1,6 @@
 # Experiments — every number in the technical report, one line each
 
-Reproduces Table 1, the Findings (§5), and Table 3 of the technical report (submitted separately). All experiments share one deterministic feature cache and
+Reproduces Table 1, the Findings (§5), and Table 3 of `report/technical_report.html`. All experiments share one deterministic feature cache and
 score against `inference/annotations.jsonl` with the frozen metric (`common.py`).
 
 ## Setup (once)

experiments/exp_encoders.py

@@ -1,84 +1,84 @@
-"""Table 3 (Section 6.2) — alternative encoders: swap WavLM-large for wav2vec2-large, HuBERT-xlarge (1B), or an
-MSP-Podcast emotion-fine-tuned wav2vec2. Each gets its own per-layer mean cache (auto-extracted on first
-run; models auto-download from the HuggingFace Hub) and is granted its BEST layer on the evaluation set —
-an upper bound for the alternative. Mean pooling for all (the comparison predates the pooling upgrades).
-
-    python experiments/exp_encoders.py           (fast if caches exist; else ~20 min per model on GPU)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import hashlib
-import numpy as np
-from experiments import common as C
-from empathyeval.data.audio import cached_load
-from empathyeval.data.release import build_index
-
-MODELS = [  # (model id, fp16, audio cap seconds) — fp16/cap12 keeps the 1B model inside 8 GB
-    ("facebook/wav2vec2-large-lv60", False, 15),
-    ("facebook/hubert-xlarge-ll60k", True, 12),
-    ("audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim", False, 15),
-]
-
-
-def cache_dir(mid):
-    return "cache/" + mid.split("/")[-1].replace("-", "") + "_L"
-
-
-def extract(mid, fp16, cap):
-    import torch
-    torch.backends.cudnn.deterministic = True
-    torch.backends.cudnn.benchmark = False
-    from transformers import AutoFeatureExtractor, AutoModel
-    dev = "cuda" if torch.cuda.is_available() else "cpu"
-    fe = AutoFeatureExtractor.from_pretrained(mid)
-    kw = {"torch_dtype": torch.float16} if fp16 else {}
-    m = AutoModel.from_pretrained(mid, **kw).to(dev).eval()
-    print(f"  extracting {mid} on {dev}", flush=True)
-    cd = cache_dir(mid); os.makedirs(cd, exist_ok=True)
-    wavs = [w for it in C.train_items(2500) for w in (it.good_wav, it.bad_wav)]
-    wavs += [o.wav for q in C.QS for o in q.options]
-    for n, w in enumerate(dict.fromkeys(wavs), 1):
-        p = os.path.join(cd, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy")
-        if os.path.exists(p):
-            continue
-        y = cached_load(w, C.cfg)[:16000 * cap]
-        inp = fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(dev)
-        if fp16:
-            inp = inp.half()
-        with torch.no_grad():
-            hs = m(inp, output_hidden_states=True).hidden_states
-        np.save(p, np.stack([h.mean(dim=1).squeeze(0).float().cpu().numpy() for h in hs]).astype(np.float32))
-        if n % 500 == 0:
-            print(f"    {n} clips", flush=True)
-
-
-def sweep(mid):
-    cd = cache_dir(mid)
-    mem = {}
-
-    def featL(L):
-        def f(w):
-            a = mem.get(w)
-            if a is None:
-                a = mem[w] = np.load(os.path.join(cd, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy"))
-            return a[L]
-        return f
-    nL = np.load(os.path.join(cd, os.listdir(cd)[0])).shape[0]
-    scores = {L: C.final(C.answers(featL(L))) for L in range(nL)}
-    best = max(scores, key=scores.get)
-    print(f"  {mid}: best layer {best}/{nL - 1} -> Final {scores[best]:.4f}  (last layer {scores[nL - 1]:.4f})", flush=True)
-
-
-print("WavLM-large reference (mean pooling):")
-C.report("wavlm-large L9 (dev-selected)", C.answers(C.feat([C.R_MEAN()])))
-_scores = {L: C.final(C.answers(C.feat([C.R_MEAN(L)]))) for L in range(25)}
-_b = max(_scores, key=_scores.get)
-print(f"  wavlm-large best layer {_b}/24 -> Final {_scores[_b]:.4f}  (symmetric post-hoc protocol)", flush=True)
-for mid, fp16, cap in MODELS:
-    wavs0 = C.train_items(1)[0].good_wav
-    probe = os.path.join(cache_dir(mid), hashlib.md5(wavs0.encode()).hexdigest()[:16] + ".npy")
-    if not os.path.exists(probe):
-        extract(mid, fp16, cap)
-    sweep(mid)
-print("(emotion2vec+: evaluated from its archived ranker predictions — see exp_ensemble.py singles)")
+"""Table 3 (Section 6.2) — alternative encoders: swap WavLM-large for wav2vec2-large, HuBERT-xlarge (1B), or an
+MSP-Podcast emotion-fine-tuned wav2vec2. Each gets its own per-layer mean cache (auto-extracted on first
+run; models auto-download from the HuggingFace Hub) and is granted its BEST layer on the evaluation set —
+an upper bound for the alternative. Mean pooling for all (the comparison predates the pooling upgrades).
+
+    python experiments/exp_encoders.py           (fast if caches exist; else ~20 min per model on GPU)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import hashlib
+import numpy as np
+from experiments import common as C
+from empathyeval.data.audio import cached_load
+from empathyeval.data.release import build_index
+
+MODELS = [  # (model id, fp16, audio cap seconds) — fp16/cap12 keeps the 1B model inside 8 GB
+    ("facebook/wav2vec2-large-lv60", False, 15),
+    ("facebook/hubert-xlarge-ll60k", True, 12),
+    ("audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim", False, 15),
+]
+
+
+def cache_dir(mid):
+    return "cache/" + mid.split("/")[-1].replace("-", "") + "_L"
+
+
+def extract(mid, fp16, cap):
+    import torch
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+    from transformers import AutoFeatureExtractor, AutoModel
+    dev = "cuda" if torch.cuda.is_available() else "cpu"
+    fe = AutoFeatureExtractor.from_pretrained(mid)
+    kw = {"torch_dtype": torch.float16} if fp16 else {}
+    m = AutoModel.from_pretrained(mid, **kw).to(dev).eval()
+    print(f"  extracting {mid} on {dev}", flush=True)
+    cd = cache_dir(mid); os.makedirs(cd, exist_ok=True)
+    wavs = [w for it in C.train_items(2500) for w in (it.good_wav, it.bad_wav)]
+    wavs += [o.wav for q in C.QS for o in q.options]
+    for n, w in enumerate(dict.fromkeys(wavs), 1):
+        p = os.path.join(cd, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy")
+        if os.path.exists(p):
+            continue
+        y = cached_load(w, C.cfg)[:16000 * cap]
+        inp = fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(dev)
+        if fp16:
+            inp = inp.half()
+        with torch.no_grad():
+            hs = m(inp, output_hidden_states=True).hidden_states
+        np.save(p, np.stack([h.mean(dim=1).squeeze(0).float().cpu().numpy() for h in hs]).astype(np.float32))
+        if n % 500 == 0:
+            print(f"    {n} clips", flush=True)
+
+
+def sweep(mid):
+    cd = cache_dir(mid)
+    mem = {}
+
+    def featL(L):
+        def f(w):
+            a = mem.get(w)
+            if a is None:
+                a = mem[w] = np.load(os.path.join(cd, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy"))
+            return a[L]
+        return f
+    nL = np.load(os.path.join(cd, os.listdir(cd)[0])).shape[0]
+    scores = {L: C.final(C.answers(featL(L))) for L in range(nL)}
+    best = max(scores, key=scores.get)
+    print(f"  {mid}: best layer {best}/{nL - 1} -> Final {scores[best]:.4f}  (last layer {scores[nL - 1]:.4f})", flush=True)
+
+
+print("WavLM-large reference (mean pooling):")
+C.report("wavlm-large L9 (dev-selected)", C.answers(C.feat([C.R_MEAN()])))
+_scores = {L: C.final(C.answers(C.feat([C.R_MEAN(L)]))) for L in range(25)}
+_b = max(_scores, key=_scores.get)
+print(f"  wavlm-large best layer {_b}/24 -> Final {_scores[_b]:.4f}  (symmetric post-hoc protocol)", flush=True)
+for mid, fp16, cap in MODELS:
+    wavs0 = C.train_items(1)[0].good_wav
+    probe = os.path.join(cache_dir(mid), hashlib.md5(wavs0.encode()).hexdigest()[:16] + ".npy")
+    if not os.path.exists(probe):
+        extract(mid, fp16, cap)
+    sweep(mid)
+print("(emotion2vec+: evaluated from its archived ranker predictions — see exp_ensemble.py singles)")

experiments/exp_ensemble.py

@@ -1,48 +1,48 @@
-"""Table 3 — judge comparison: individual scores of the three archived judges (Qwen3-Omni-30B textualized
-judge, Voxtral-24B omni judge, emotion2vec ranker), plus — for completeness — the uniform majority-vote
-combinations we explored and rejected during development.
-
-The judges' per-question predictions are shipped as artifacts (experiments/artifacts/answers_*.json):
-re-running those 24-30B models is expensive and, for the textualized judge, involved development-time API
-access — the archived predictions make the comparison exactly reproducible. The WavLM member is retrained
-live from the unified cache (mean pooling, matching the stage at which the comparison was run).
-Ties break to the strongest member in canonical order wav > cv3 > vox > emb. No dataset-label branching.
-
-    python experiments/exp_ensemble.py          (~2 min)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import json
-from collections import Counter
-from itertools import combinations
-from experiments import common as C
-
-ART = os.path.join(os.path.dirname(os.path.abspath(__file__)), "artifacts")
-preds = {"wav": C.answers(C.feat([C.R_MEAN()]))}
-for k, f in [("cv3", "answers_qwen3omni.json"), ("vox", "answers_voxtral.json"), ("emb", "answers_emotion2vec.json")]:
-    preds[k] = json.load(open(os.path.join(ART, f), encoding="utf-8"))
-ORDER = ["wav", "cv3", "vox", "emb"]
-
-
-def vote(keys):
-    out = {}
-    for q in C.QS:
-        letters = [o.letter for o in q.options]
-        counts = Counter(preds[k][q.qid] for k in keys if q.qid in preds[k])
-        if not counts:
-            out[q.qid] = letters[0]; continue
-        top = counts.most_common(1)[0][1]
-        tied = {a for a, n in counts.items() if n == top}
-        out[q.qid] = next(iter(tied)) if len(tied) == 1 else \
-            next((preds[k][q.qid] for k in keys if k in ORDER and preds[k].get(q.qid) in tied), letters[0])
-    return out
-
-
-print("== singles ==")
-for k in ORDER:
-    C.report(k, preds[k])
-print("== pairs / trios / full (uniform vote, tie -> strongest) ==")
-for r in (2, 3, 4):
-    for combo in combinations(ORDER, r):
-        C.report("+".join(combo), vote(list(combo)))
+"""Table 3 — judge comparison: individual scores of the three archived judges (Qwen3-Omni-30B textualized
+judge, Voxtral-24B omni judge, emotion2vec ranker), plus — for completeness — the uniform majority-vote
+combinations we explored and rejected during development.
+
+The judges' per-question predictions are shipped as artifacts (experiments/artifacts/answers_*.json):
+re-running those 24-30B models is expensive and, for the textualized judge, involved development-time API
+access — the archived predictions make the comparison exactly reproducible. The WavLM member is retrained
+live from the unified cache (mean pooling, matching the stage at which the comparison was run).
+Ties break to the strongest member in canonical order wav > cv3 > vox > emb. No dataset-label branching.
+
+    python experiments/exp_ensemble.py          (~2 min)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import json
+from collections import Counter
+from itertools import combinations
+from experiments import common as C
+
+ART = os.path.join(os.path.dirname(os.path.abspath(__file__)), "artifacts")
+preds = {"wav": C.answers(C.feat([C.R_MEAN()]))}
+for k, f in [("cv3", "answers_qwen3omni.json"), ("vox", "answers_voxtral.json"), ("emb", "answers_emotion2vec.json")]:
+    preds[k] = json.load(open(os.path.join(ART, f), encoding="utf-8"))
+ORDER = ["wav", "cv3", "vox", "emb"]
+
+
+def vote(keys):
+    out = {}
+    for q in C.QS:
+        letters = [o.letter for o in q.options]
+        counts = Counter(preds[k][q.qid] for k in keys if q.qid in preds[k])
+        if not counts:
+            out[q.qid] = letters[0]; continue
+        top = counts.most_common(1)[0][1]
+        tied = {a for a, n in counts.items() if n == top}
+        out[q.qid] = next(iter(tied)) if len(tied) == 1 else \
+            next((preds[k][q.qid] for k in keys if k in ORDER and preds[k].get(q.qid) in tied), letters[0])
+    return out
+
+
+print("== singles ==")
+for k in ORDER:
+    C.report(k, preds[k])
+print("== pairs / trios / full (uniform vote, tie -> strongest) ==")
+for r in (2, 3, 4):
+    for combo in combinations(ORDER, r):
+        C.report("+".join(combo), vote(list(combo)))

experiments/exp_finetune.py

@@ -1,95 +1,95 @@
-"""Table 1 fine-tuning row / Finding 3 — fine-tuning WavLM on the pairwise task (a negative result: never beats frozen features).
-Truncates WavLM to 9 layers, unfreezes the top 3 + a linear scoring head, and trains the pairwise
-good>bad objective directly on audio. LONG-RUNNING (hours on an 8 GB GPU).
-
-    python experiments/exp_finetune.py warm      # head warm-started from the frozen-feature solution
-    python experiments/exp_finetune.py random    # randomly initialized head
-    python experiments/exp_finetune.py gentle    # warm start + 10x lower learning rate
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import numpy as np
-import torch
-import torch.nn as nn
-from experiments import common as C
-from empathyeval.data.audio import cached_load
-
-MODE = sys.argv[1] if len(sys.argv) > 1 else "warm"
-LR = 1e-6 if MODE == "gentle" else 1e-5
-STEPS, EVAL_EVERY, KEEP, UNFREEZE = 3000, 300, 9, 3
-DEV = "cuda" if torch.cuda.is_available() else "cpu"
-
-
-class Ranker(nn.Module):
-    def __init__(self):
-        super().__init__()
-        from transformers import WavLMModel
-        m = WavLMModel.from_pretrained(C.MID)
-        m.encoder.layers = m.encoder.layers[:KEEP]           # last_hidden_state == layer-9 output
-        m.gradient_checkpointing_enable()
-        for p in m.parameters():
-            p.requires_grad_(False)
-        for lyr in m.encoder.layers[KEEP - UNFREEZE:]:
-            for p in lyr.parameters():
-                p.requires_grad_(True)
-        self.wavlm, self.head = m, nn.Linear(1024, 1)
-
-    def score(self, iv):
-        return self.head(self.wavlm(iv).last_hidden_state.mean(dim=1)).squeeze(-1)
-
-
-def warm_start(model):
-    """Init the head from the frozen layer-9 mean-pool logistic, so training starts AT the frozen baseline."""
-    sc, clf = C.train_ranker(C.feat([C.R_MEAN()]))
-    w, b, mu, sd = clf.coef_[0], clf.intercept_[0], sc.mean_, sc.scale_
-    with torch.no_grad():
-        model.head.weight.copy_(torch.tensor((w / sd)[None, :], dtype=model.head.weight.dtype))
-        model.head.bias.copy_(torch.tensor(b - float((w * mu / sd).sum()), dtype=model.head.bias.dtype))
-
-
-from transformers import AutoFeatureExtractor  # noqa: E402
-fe = AutoFeatureExtractor.from_pretrained(C.MID)
-
-
-def inp(w):
-    y = cached_load(w, C.cfg)[:16000 * C.CAP_S]
-    return fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(DEV)
-
-
-def evaluate(model):
-    model.eval()
-    ans = {}
-    with torch.no_grad():
-        for q in C.QS:
-            s = [model.score(inp(o.wav)).item() for o in q.options]
-            ans[q.qid] = [o.letter for o in q.options][int(np.argmax(s))]
-    model.train()
-    return ans
-
-
-model = Ranker().to(DEV).train()
-if MODE in ("warm", "gentle"):
-    warm_start(model)
-opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=LR)
-scaler = torch.cuda.amp.GradScaler(enabled=DEV == "cuda")
-bce = nn.BCEWithLogitsLoss()
-items = C.train_items(2500)
-print(f"mode={MODE} lr={LR}  frozen baseline (mean pooling): {C.final(C.answers(C.feat([C.R_MEAN()]))):.4f}")
-best = -1.0
-for step in range(1, STEPS + 1):
-    it = items[(step - 1) % len(items)]
-    try:
-        gi, bi = inp(it.good_wav), inp(it.bad_wav)
-    except Exception:
-        continue
-    with torch.autocast(DEV, enabled=DEV == "cuda"):
-        loss = bce(model.score(gi) - model.score(bi), torch.ones(1, device=DEV))
-    opt.zero_grad(); scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
-    if step % 50 == 0:
-        print(f"step {step}/{STEPS} loss {loss.item():.3f}", flush=True)
-    if step % EVAL_EVERY == 0:
-        f = C.final(evaluate(model))
-        best = max(best, f)
-        print(f"  [eval] step {step}: Final={f:.4f} (best {best:.4f})", flush=True)
-print(f"DONE mode={MODE}: best fine-tuned Final {best:.4f}")
+"""Table 1 fine-tuning row / Finding 3 — fine-tuning WavLM on the pairwise task (a negative result: never beats frozen features).
+Truncates WavLM to 9 layers, unfreezes the top 3 + a linear scoring head, and trains the pairwise
+good>bad objective directly on audio. LONG-RUNNING (hours on an 8 GB GPU).
+
+    python experiments/exp_finetune.py warm      # head warm-started from the frozen-feature solution
+    python experiments/exp_finetune.py random    # randomly initialized head
+    python experiments/exp_finetune.py gentle    # warm start + 10x lower learning rate
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import numpy as np
+import torch
+import torch.nn as nn
+from experiments import common as C
+from empathyeval.data.audio import cached_load
+
+MODE = sys.argv[1] if len(sys.argv) > 1 else "warm"
+LR = 1e-6 if MODE == "gentle" else 1e-5
+STEPS, EVAL_EVERY, KEEP, UNFREEZE = 3000, 300, 9, 3
+DEV = "cuda" if torch.cuda.is_available() else "cpu"
+
+
+class Ranker(nn.Module):
+    def __init__(self):
+        super().__init__()
+        from transformers import WavLMModel
+        m = WavLMModel.from_pretrained(C.MID)
+        m.encoder.layers = m.encoder.layers[:KEEP]           # last_hidden_state == layer-9 output
+        m.gradient_checkpointing_enable()
+        for p in m.parameters():
+            p.requires_grad_(False)
+        for lyr in m.encoder.layers[KEEP - UNFREEZE:]:
+            for p in lyr.parameters():
+                p.requires_grad_(True)
+        self.wavlm, self.head = m, nn.Linear(1024, 1)
+
+    def score(self, iv):
+        return self.head(self.wavlm(iv).last_hidden_state.mean(dim=1)).squeeze(-1)
+
+
+def warm_start(model):
+    """Init the head from the frozen layer-9 mean-pool logistic, so training starts AT the frozen baseline."""
+    sc, clf = C.train_ranker(C.feat([C.R_MEAN()]))
+    w, b, mu, sd = clf.coef_[0], clf.intercept_[0], sc.mean_, sc.scale_
+    with torch.no_grad():
+        model.head.weight.copy_(torch.tensor((w / sd)[None, :], dtype=model.head.weight.dtype))
+        model.head.bias.copy_(torch.tensor(b - float((w * mu / sd).sum()), dtype=model.head.bias.dtype))
+
+
+from transformers import AutoFeatureExtractor  # noqa: E402
+fe = AutoFeatureExtractor.from_pretrained(C.MID)
+
+
+def inp(w):
+    y = cached_load(w, C.cfg)[:16000 * C.CAP_S]
+    return fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(DEV)
+
+
+def evaluate(model):
+    model.eval()
+    ans = {}
+    with torch.no_grad():
+        for q in C.QS:
+            s = [model.score(inp(o.wav)).item() for o in q.options]
+            ans[q.qid] = [o.letter for o in q.options][int(np.argmax(s))]
+    model.train()
+    return ans
+
+
+model = Ranker().to(DEV).train()
+if MODE in ("warm", "gentle"):
+    warm_start(model)
+opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=LR)
+scaler = torch.cuda.amp.GradScaler(enabled=DEV == "cuda")
+bce = nn.BCEWithLogitsLoss()
+items = C.train_items(2500)
+print(f"mode={MODE} lr={LR}  frozen baseline (mean pooling): {C.final(C.answers(C.feat([C.R_MEAN()]))):.4f}")
+best = -1.0
+for step in range(1, STEPS + 1):
+    it = items[(step - 1) % len(items)]
+    try:
+        gi, bi = inp(it.good_wav), inp(it.bad_wav)
+    except Exception:
+        continue
+    with torch.autocast(DEV, enabled=DEV == "cuda"):
+        loss = bce(model.score(gi) - model.score(bi), torch.ones(1, device=DEV))
+    opt.zero_grad(); scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
+    if step % 50 == 0:
+        print(f"step {step}/{STEPS} loss {loss.item():.3f}", flush=True)
+    if step % EVAL_EVERY == 0:
+        f = C.final(evaluate(model))
+        best = max(best, f)
+        print(f"  [eval] step {step}: Final={f:.4f} (best {best:.4f})", flush=True)
+print(f"DONE mode={MODE}: best fine-tuned Final {best:.4f}")

experiments/exp_heads.py

@@ -1,44 +1,44 @@
-"""Table 1 ranker-head row / Finding 5 — ranker heads on the final pooling features: logistic C sweep, an MLP, and an RBF-SVM.
-
-    python experiments/exp_heads.py              (~20 min; the RBF-SVM on 5120-d features dominates)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import numpy as np
-from sklearn.linear_model import LogisticRegression
-from sklearn.neural_network import MLPClassifier
-from sklearn.preprocessing import StandardScaler
-from sklearn.svm import SVC
-from experiments import common as C
-
-ff = C.feat(C.POOL_FINAL)
-X, y = [], []
-for it in C.train_items(2500):
-    try:
-        g, b = ff(it.good_wav), ff(it.bad_wav)
-    except Exception:
-        continue
-    X.append(g - b); y.append(1)
-    X.append(b - g); y.append(0)
-X = np.array(X)
-sc = StandardScaler().fit(X)
-Xs = sc.transform(X)
-
-
-def run(name, clf, proba=False):
-    clf.fit(Xs, y)
-    fn = (lambda z: clf.predict_proba(z)[:, 1]) if proba else clf.decision_function
-    ans = {}
-    for q in C.QS:
-        fv = sc.transform([ff(o.wav) for o in q.options])
-        ans[q.qid] = [o.letter for o in q.options][int(np.argmax(fn(fv)))]
-    C.report(name, ans)
-
-
-for Creg in [0.1, 0.25, 0.5, 1.0, 2.0]:
-    run(f"LogReg C={Creg}", LogisticRegression(max_iter=3000, C=Creg))
-run("MLP(64)", MLPClassifier(hidden_layer_sizes=(64,), alpha=1e-2, max_iter=500,
-                             early_stopping=True, random_state=0), proba=True)
-print("  (RBF-SVM is slow on 5120-d features ...)", flush=True)
-run("RBF-SVM C=1", SVC(C=1.0, kernel="rbf", gamma="scale"))
+"""Table 1 ranker-head row / Finding 5 — ranker heads on the final pooling features: logistic C sweep, an MLP, and an RBF-SVM.
+
+    python experiments/exp_heads.py              (~20 min; the RBF-SVM on 5120-d features dominates)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.neural_network import MLPClassifier
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import SVC
+from experiments import common as C
+
+ff = C.feat(C.POOL_FINAL)
+X, y = [], []
+for it in C.train_items(2500):
+    try:
+        g, b = ff(it.good_wav), ff(it.bad_wav)
+    except Exception:
+        continue
+    X.append(g - b); y.append(1)
+    X.append(b - g); y.append(0)
+X = np.array(X)
+sc = StandardScaler().fit(X)
+Xs = sc.transform(X)
+
+
+def run(name, clf, proba=False):
+    clf.fit(Xs, y)
+    fn = (lambda z: clf.predict_proba(z)[:, 1]) if proba else clf.decision_function
+    ans = {}
+    for q in C.QS:
+        fv = sc.transform([ff(o.wav) for o in q.options])
+        ans[q.qid] = [o.letter for o in q.options][int(np.argmax(fn(fv)))]
+    C.report(name, ans)
+
+
+for Creg in [0.1, 0.25, 0.5, 1.0, 2.0]:
+    run(f"LogReg C={Creg}", LogisticRegression(max_iter=3000, C=Creg))
+run("MLP(64)", MLPClassifier(hidden_layer_sizes=(64,), alpha=1e-2, max_iter=500,
+                             early_stopping=True, random_state=0), proba=True)
+print("  (RBF-SVM is slow on 5120-d features ...)", flush=True)
+run("RBF-SVM C=1", SVC(C=1.0, kernel="rbf", gamma="scale"))

experiments/exp_layers.py

@@ -1,20 +1,20 @@
-"""Table 1 layer row / Finding 2 — layer sweep: Final per WavLM hidden layer, under mean pooling and mean+std pooling.
-
-    python experiments/exp_layers.py            (~5 min: trains 50 rankers)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-from experiments import common as C
-
-print("layer   mean-Final   mean+std-Final")
-best = {}
-for L in range(25):
-    fm = C.final(C.answers(C.feat([C.R_MEAN(L)])))
-    fs = C.final(C.answers(C.feat([C.R_MEAN(L), C.R_STD(L)])))
-    best[L] = (fm, fs)
-    print(f"  {L:2d}     {fm:.4f}       {fs:.4f}", flush=True)
-bm = max(best, key=lambda L: best[L][0])
-bs = max(best, key=lambda L: best[L][1])
-print(f"best layer (mean): {bm} ({best[bm][0]:.4f})   last layer: {best[24][0]:.4f}")
-print(f"best layer (mean+std): {bs} ({best[bs][1]:.4f})")
+"""Table 1 layer row / Finding 2 — layer sweep: Final per WavLM hidden layer, under mean pooling and mean+std pooling.
+
+    python experiments/exp_layers.py            (~5 min: trains 50 rankers)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+from experiments import common as C
+
+print("layer   mean-Final   mean+std-Final")
+best = {}
+for L in range(25):
+    fm = C.final(C.answers(C.feat([C.R_MEAN(L)])))
+    fs = C.final(C.answers(C.feat([C.R_MEAN(L), C.R_STD(L)])))
+    best[L] = (fm, fs)
+    print(f"  {L:2d}     {fm:.4f}       {fs:.4f}", flush=True)
+bm = max(best, key=lambda L: best[L][0])
+bs = max(best, key=lambda L: best[L][1])
+print(f"best layer (mean): {bm} ({best[bm][0]:.4f})   last layer: {best[24][0]:.4f}")
+print(f"best layer (mean+std): {bs} ({best[bs][1]:.4f})")

experiments/exp_pooling.py

@@ -1,36 +1,36 @@
-"""Table 1 pooling & K rows / Finding 1 — pooling ablation: mean -> +std -> +3 segment means (with paired bootstraps), plus the
-rejected alternatives (max / velocity / delta-std / L2 norm / multi-layer / speech-segments / K sweep).
-
-    python experiments/exp_pooling.py           (~10 min; bootstraps dominate)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import numpy as np
-from experiments import common as C
-
-MEAN = [C.R_MEAN()]
-MS = [C.R_MEAN(), C.R_STD()]
-
-print("== Table 2: pooling ablation (WavLM layer 9) ==")
-a_mean = C.answers(C.feat(MEAN)); C.report("mean", a_mean)
-a_ms = C.answers(C.feat(MS)); C.report("mean+std", a_ms)
-a_fin = C.answers(C.feat(C.POOL_FINAL)); C.report("mean+std+3seg (final)", a_fin)
-b1 = C.paired_bootstrap(a_mean, a_ms)
-b2 = C.paired_bootstrap(a_ms, a_fin)
-print(f"  bootstrap mean->mean+std : median {b1['median']:+.4f}  95% CI [{b1['lo']:+.4f},{b1['hi']:+.4f}]  P(better)={b1['p_better']:.1%}")
-print(f"  bootstrap +std->final    : median {b2['median']:+.4f}  95% CI [{b2['lo']:+.4f},{b2['hi']:+.4f}]  P(better)={b2['p_better']:.1%}")
-
-print("== rejected alternatives ==")
-C.report("mean+std+max", C.answers(C.feat(MS + [C.R_MAX])))
-C.report("mean+std+velocity", C.answers(C.feat(MS + [C.R_DVEL])))
-C.report("mean+std+delta-std", C.answers(C.feat(MS + [C.R_DSTD])))
-_ff = C.feat(C.POOL_FINAL)
-C.report("final, L2-normalized", C.answers(lambda w: (lambda v: v / (np.linalg.norm(v) + 1e-8))(_ff(w))))
-C.report("multi-layer L8+9+10 mean+std", C.answers(C.feat(
-    [C.R_MEAN(8), C.R_STD(8), C.R_MEAN(9), C.R_STD(9), C.R_MEAN(10), C.R_STD(10)])))
-C.report("mean+std+3 speech-seg (VAD)", C.answers(C.feat(MS + [C.R_SPEECH0, C.R_SPEECH0 + 1, C.R_SPEECH0 + 2])))
-
-print("== segment-count sweep (mean+std + K derived equal-time segments) ==")
-for K in [2, 3, 4, 6, 12]:
-    C.report(f"K={K}", C.answers(C.feat_fineK(K)))
+"""Table 1 pooling & K rows / Finding 1 — pooling ablation: mean -> +std -> +3 segment means (with paired bootstraps), plus the
+rejected alternatives (max / velocity / delta-std / L2 norm / multi-layer / speech-segments / K sweep).
+
+    python experiments/exp_pooling.py           (~10 min; bootstraps dominate)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import numpy as np
+from experiments import common as C
+
+MEAN = [C.R_MEAN()]
+MS = [C.R_MEAN(), C.R_STD()]
+
+print("== Table 2: pooling ablation (WavLM layer 9) ==")
+a_mean = C.answers(C.feat(MEAN)); C.report("mean", a_mean)
+a_ms = C.answers(C.feat(MS)); C.report("mean+std", a_ms)
+a_fin = C.answers(C.feat(C.POOL_FINAL)); C.report("mean+std+3seg (final)", a_fin)
+b1 = C.paired_bootstrap(a_mean, a_ms)
+b2 = C.paired_bootstrap(a_ms, a_fin)
+print(f"  bootstrap mean->mean+std : median {b1['median']:+.4f}  95% CI [{b1['lo']:+.4f},{b1['hi']:+.4f}]  P(better)={b1['p_better']:.1%}")
+print(f"  bootstrap +std->final    : median {b2['median']:+.4f}  95% CI [{b2['lo']:+.4f},{b2['hi']:+.4f}]  P(better)={b2['p_better']:.1%}")
+
+print("== rejected alternatives ==")
+C.report("mean+std+max", C.answers(C.feat(MS + [C.R_MAX])))
+C.report("mean+std+velocity", C.answers(C.feat(MS + [C.R_DVEL])))
+C.report("mean+std+delta-std", C.answers(C.feat(MS + [C.R_DSTD])))
+_ff = C.feat(C.POOL_FINAL)
+C.report("final, L2-normalized", C.answers(lambda w: (lambda v: v / (np.linalg.norm(v) + 1e-8))(_ff(w))))
+C.report("multi-layer L8+9+10 mean+std", C.answers(C.feat(
+    [C.R_MEAN(8), C.R_STD(8), C.R_MEAN(9), C.R_STD(9), C.R_MEAN(10), C.R_STD(10)])))
+C.report("mean+std+3 speech-seg (VAD)", C.answers(C.feat(MS + [C.R_SPEECH0, C.R_SPEECH0 + 1, C.R_SPEECH0 + 2])))
+
+print("== segment-count sweep (mean+std + K derived equal-time segments) ==")
+for K in [2, 3, 4, 6, 12]:
+    C.report(f"K={K}", C.answers(C.feat_fineK(K)))

experiments/exp_trainsize.py

@@ -1,11 +1,11 @@
-"""Table 1 training-size row / Finding 3 — training-set size: Final vs. number of training triples (final pooling, seed-0 order).
-
-    python experiments/exp_trainsize.py          (~3 min)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-from experiments import common as C
-
-for n in [500, 1000, 1500, 2000, 2500, 3000, 4000, 4892]:
-    C.report(f"N={n}", C.answers(C.feat(C.POOL_FINAL), n_items=n))
+"""Table 1 training-size row / Finding 3 — training-set size: Final vs. number of training triples (final pooling, seed-0 order).
+
+    python experiments/exp_trainsize.py          (~3 min)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+from experiments import common as C
+
+for n in [500, 1000, 1500, 2000, 2500, 3000, 4000, 4892]:
+    C.report(f"N={n}", C.answers(C.feat(C.POOL_FINAL), n_items=n))

experiments/exp_utterance.py

@@ -1,50 +1,50 @@
-"""Table 1 utterance-conditioning row / Finding 4 — utterance conditioning: inject the human utterance's acoustics via interaction features
-(cosine delivery-match / element-wise products). Plain concatenation cancels in the pairwise difference,
-so interactions are the only way the utterance can enter a linear pairwise ranker.
-
-    python experiments/exp_utterance.py          (~3 min)
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import numpy as np
-from sklearn.linear_model import LogisticRegression
-from sklearn.preprocessing import StandardScaler
-from experiments import common as C
-
-
-def nrm(v):
-    return v / (np.linalg.norm(v) + 1e-8)
-
-
-def feature(resp, utt, mode):
-    a = C.arr(resp)
-    base = np.concatenate([a[r] for r in C.POOL_FINAL])
-    if mode == "base":
-        return base
-    um = C.arr(utt)[C.R_MEAN()]
-    resp_parts = [a[C.R_MEAN()], a[C.R_SEG3], a[C.R_SEG3 + 1], a[C.R_SEG3 + 2]]
-    if mode == "+cos":
-        return np.concatenate([base, np.array([float(nrm(p) @ nrm(um)) for p in resp_parts], dtype=np.float32)])
-    if mode == "+prod":
-        return np.concatenate([base, a[C.R_MEAN()] * um])
-    raise ValueError(mode)
-
-
-for mode in ["base", "+cos", "+prod"]:
-    X, y = [], []
-    for it in C.train_items(2500):
-        try:
-            g, b = feature(it.good_wav, it.utterance_wav, mode), feature(it.bad_wav, it.utterance_wav, mode)
-        except Exception:
-            continue
-        X.append(g - b); y.append(1)
-        X.append(b - g); y.append(0)
-    X = np.array(X)
-    sc = StandardScaler().fit(X)
-    clf = LogisticRegression(max_iter=3000, C=0.5).fit(sc.transform(X), y)
-    ans = {}
-    for q in C.QS:
-        fv = sc.transform([feature(o.wav, q.utterance_wav, mode) for o in q.options])
-        ans[q.qid] = [o.letter for o in q.options][int(np.argmax(clf.decision_function(fv)))]
-    C.report(mode, ans)
+"""Table 1 utterance-conditioning row / Finding 4 — utterance conditioning: inject the human utterance's acoustics via interaction features
+(cosine delivery-match / element-wise products). Plain concatenation cancels in the pairwise difference,
+so interactions are the only way the utterance can enter a linear pairwise ranker.
+
+    python experiments/exp_utterance.py          (~3 min)
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.preprocessing import StandardScaler
+from experiments import common as C
+
+
+def nrm(v):
+    return v / (np.linalg.norm(v) + 1e-8)
+
+
+def feature(resp, utt, mode):
+    a = C.arr(resp)
+    base = np.concatenate([a[r] for r in C.POOL_FINAL])
+    if mode == "base":
+        return base
+    um = C.arr(utt)[C.R_MEAN()]
+    resp_parts = [a[C.R_MEAN()], a[C.R_SEG3], a[C.R_SEG3 + 1], a[C.R_SEG3 + 2]]
+    if mode == "+cos":
+        return np.concatenate([base, np.array([float(nrm(p) @ nrm(um)) for p in resp_parts], dtype=np.float32)])
+    if mode == "+prod":
+        return np.concatenate([base, a[C.R_MEAN()] * um])
+    raise ValueError(mode)
+
+
+for mode in ["base", "+cos", "+prod"]:
+    X, y = [], []
+    for it in C.train_items(2500):
+        try:
+            g, b = feature(it.good_wav, it.utterance_wav, mode), feature(it.bad_wav, it.utterance_wav, mode)
+        except Exception:
+            continue
+        X.append(g - b); y.append(1)
+        X.append(b - g); y.append(0)
+    X = np.array(X)
+    sc = StandardScaler().fit(X)
+    clf = LogisticRegression(max_iter=3000, C=0.5).fit(sc.transform(X), y)
+    ans = {}
+    for q in C.QS:
+        fv = sc.transform([feature(o.wav, q.utterance_wav, mode) for o in q.options])
+        ans[q.qid] = [o.letter for o in q.options][int(np.argmax(clf.decision_function(fv)))]
+    C.report(mode, ans)

experiments/exp_window.py

@@ -1,58 +1,58 @@
-"""Table 1 context-window row / Finding 4 — context-window length: mean-pooled layer-9 performance with 15/30/45/60 s of audio.
-Uses a 9-layer-truncated WavLM (last_hidden_state == layer-9 output) so 60 s fits an 8 GB GPU, and one
-60 s forward per clip whose prefix means give every shorter cap. Own cache (~35 min extraction on GPU).
-
-    python experiments/exp_window.py
-"""
-import sys
-import os
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
-import hashlib
-import numpy as np
-from experiments import common as C
-from empathyeval.data.audio import cached_load
-
-CAPS = [15, 30, 45, 60]
-CD = "cache/exp_window"
-os.makedirs(CD, exist_ok=True)
-
-
-def cp(w):
-    return os.path.join(CD, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy")
-
-
-def extract():
-    import torch
-    torch.backends.cudnn.deterministic = True
-    torch.backends.cudnn.benchmark = False
-    from transformers import AutoFeatureExtractor, WavLMModel
-    dev = "cuda" if torch.cuda.is_available() else "cpu"
-    fe = AutoFeatureExtractor.from_pretrained(C.MID)
-    m = WavLMModel.from_pretrained(C.MID)
-    m.encoder.layers = m.encoder.layers[:C.LAYER]          # truncate -> output == layer-9
-    m = m.to(dev).eval()
-    print(f"extracting 60 s layer-9 features on {dev}", flush=True)
-    wavs = [w for it in C.train_items(2500) for w in (it.good_wav, it.bad_wav)]
-    wavs += [o.wav for q in C.QS for o in q.options]
-    for n, w in enumerate(dict.fromkeys(wavs), 1):
-        if os.path.exists(cp(w)):
-            continue
-        y = cached_load(w, C.cfg)[:16000 * CAPS[-1]]
-        inp = fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(dev)
-        with torch.no_grad():
-            h = m(inp).last_hidden_state[0]                # [T,1024]
-        np.save(cp(w), np.stack([h[:50 * s].mean(0).cpu().numpy() for s in CAPS]).astype(np.float32))
-        if n % 500 == 0:
-            print(f"  {n} clips", flush=True)
-
-
-if not os.path.exists(cp(C.train_items(1)[0].good_wav)):
-    extract()
-mem = {}
-for i, s in enumerate(CAPS):
-    def f(w, i=i):
-        a = mem.get(w)
-        if a is None:
-            a = mem[w] = np.load(cp(w))
-        return a[i]
-    C.report(f"cap {s:2d}s (mean pooling)", C.answers(f))
+"""Table 1 context-window row / Finding 4 — context-window length: mean-pooled layer-9 performance with 15/30/45/60 s of audio.
+Uses a 9-layer-truncated WavLM (last_hidden_state == layer-9 output) so 60 s fits an 8 GB GPU, and one
+60 s forward per clip whose prefix means give every shorter cap. Own cache (~35 min extraction on GPU).
+
+    python experiments/exp_window.py
+"""
+import sys
+import os
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import hashlib
+import numpy as np
+from experiments import common as C
+from empathyeval.data.audio import cached_load
+
+CAPS = [15, 30, 45, 60]
+CD = "cache/exp_window"
+os.makedirs(CD, exist_ok=True)
+
+
+def cp(w):
+    return os.path.join(CD, hashlib.md5(w.encode()).hexdigest()[:16] + ".npy")
+
+
+def extract():
+    import torch
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+    from transformers import AutoFeatureExtractor, WavLMModel
+    dev = "cuda" if torch.cuda.is_available() else "cpu"
+    fe = AutoFeatureExtractor.from_pretrained(C.MID)
+    m = WavLMModel.from_pretrained(C.MID)
+    m.encoder.layers = m.encoder.layers[:C.LAYER]          # truncate -> output == layer-9
+    m = m.to(dev).eval()
+    print(f"extracting 60 s layer-9 features on {dev}", flush=True)
+    wavs = [w for it in C.train_items(2500) for w in (it.good_wav, it.bad_wav)]
+    wavs += [o.wav for q in C.QS for o in q.options]
+    for n, w in enumerate(dict.fromkeys(wavs), 1):
+        if os.path.exists(cp(w)):
+            continue
+        y = cached_load(w, C.cfg)[:16000 * CAPS[-1]]
+        inp = fe(y, sampling_rate=16000, return_tensors="pt").input_values.to(dev)
+        with torch.no_grad():
+            h = m(inp).last_hidden_state[0]                # [T,1024]
+        np.save(cp(w), np.stack([h[:50 * s].mean(0).cpu().numpy() for s in CAPS]).astype(np.float32))
+        if n % 500 == 0:
+            print(f"  {n} clips", flush=True)
+
+
+if not os.path.exists(cp(C.train_items(1)[0].good_wav)):
+    extract()
+mem = {}
+for i, s in enumerate(CAPS):
+    def f(w, i=i):
+        a = mem.get(w)
+        if a is None:
+            a = mem[w] = np.load(cp(w))
+        return a[i]
+    C.report(f"cap {s:2d}s (mean pooling)", C.answers(f))

inference/README.md

@@ -20,7 +20,7 @@ Self-contained. `run.py` reads the test release JSON file(s) and writes **`outpu
 ## Step-by-step
 
 ### 1. Prerequisites
-- **Python 3.10+** (the pinned `scikit-learn==1.7.2` that loads `ranker.pkl` requires ≥3.10; tested on 3.10.8)
+- **Python 3.9+**
 - **~1.5 GB free disk** for the WavLM model (downloaded automatically on first run).
 - A CUDA **GPU** is used automatically if present; otherwise it runs on **CPU** (slower, still fine).
 
@@ -28,12 +28,6 @@ Self-contained. `run.py` reads the test release JSON file(s) and writes **`outpu
 ```bash
 pip install -r requirements.txt
 ```
-For GPU, install the CUDA torch build **first** (a plain install pulls the CPU-only wheel; if the CPU
-wheel is already installed, add `--force-reinstall`):
-```bash
-pip install torch --index-url https://download.pytorch.org/whl/cu128
-pip install -r requirements.txt
-```
 
 ### 3. Point it at your test data
 Open `config.yaml` and list your test release JSON file(s):
@@ -50,9 +44,6 @@ that JSON's own folder (absolute paths also work), so keep each release file nex
 ```bash
 python run.py
 ```
-Any **relative** paths under `releases:` resolve against your *current working directory* — with the
-absolute paths of step 3 you can run from anywhere; with relative paths, run from the folder they are
-relative to.
 **On the very first run** it downloads `microsoft/wavlm-large` (~1.2 GB, MIT, open weights) from Hugging Face —
 this needs **internet once**. It is cached under `~/.cache/huggingface`, so **every later run is fully offline.**
 You'll see progress like:

inference/config.yaml

@@ -1,4 +1,4 @@
 releases:
-  - "../data/phase1-test_multi-context_gigaspeech/phase1-test_gigaspeech_release.json"
-  - "../data/phase1-test_multi-context_meld/phase1-test_meld_release.json"
-  - "../data/phase1-test_multi-emotion_emovdb/phase1-test_emovdb_release.json"
+  - "C:/Users/andy/Documents/Claude/Projects/HumOmni Track 1/humomni-empathyeval/data/phase2-test_multi-context_gigaspeech/phase2-test_gigaspeech_release.json"
+  - "C:/Users/andy/Documents/Claude/Projects/HumOmni Track 1/humomni-empathyeval/data/phase2-test_multi-context_meld/phase2-test_meld_release.json"
+  - "C:/Users/andy/Documents/Claude/Projects/HumOmni Track 1/humomni-empathyeval/data/phase2-test_multi-emotion_emovdb/phase2-test_emovdb_release.json"

inference/phase2_run.err

@@ -0,0 +1 @@
+

inference/phase2_run.out

@@ -0,0 +1,23 @@
+==================================================================
+  HumOmni 2026 Track 1 — EmpathyEval  |  inference (open weights, offline)
+==================================================================
+  loading microsoft/wavlm-large on cuda (first run downloads ~1.2 GB from Hugging Face)...
+  model ready on cuda.
+  3 release file(s), 542 questions to answer.
+
+  scoring responses with WavLM segment-pooling...
+  50/542 answered
+  100/542 answered
+  150/542 answered
+  200/542 answered
+  250/542 answered
+  300/542 answered
+  350/542 answered
+  400/542 answered
+  450/542 answered
+  500/542 answered
+  542/542 answered
+==================================================================
+  DONE — wrote 542 answers to:
+    C:\Users\andy\Documents\Claude\Projects\HumOmni Track 1\humomni-empathyeval\inference\output.jsonl
+==================================================================

inference/requirements.txt

@@ -6,7 +6,6 @@
 # (PyPI default). For GPU, install the CUDA build FIRST, then install the rest, e.g.:
 #     pip install torch --index-url https://download.pytorch.org/whl/cu128
 #     pip install -r inference/requirements.txt
-# (If a CPU-only torch is ALREADY installed, pip will skip the CUDA build — add --force-reinstall.)
 # The code auto-selects the GPU when a CUDA torch is present; CPU works too, just slower.
 torch>=2.2                # tested: 2.11.0+cu128 (CUDA); CPU wheel also works
 transformers>=4.40        # tested: 5.12.1  (microsoft/wavlm-large, downloaded once, open weights)

inference/run.py

@@ -1,6 +1,6 @@
 """INFERENCE — read the test release + the pre-trained ranker, write output.jsonl. Nothing else.
 
-    python inference/run.py        # from the repo root; relative paths in config.yaml resolve against the cwd
+    python run.py
 
 Self-contained: this file + ranker.pkl + config.yaml are all the code/model you need. It does NOT train
 and does NOT evaluate. Open weights, offline, no API, and the SAME decision for every question (it never
@@ -73,9 +73,12 @@ def read_questions(releases):
         for r in rows:
             try:
                 opts = [(k[-1], os.path.join(base, v)) for k, v in sorted(r["options"].items())]
-                qs.append((r["question_id"], opts))
+                qid = r.get("question_id") or r.get("data_index")   # Phase 2 renamed the field to data_index
+                if not qid:
+                    raise KeyError("question_id / data_index")
+                qs.append((qid, opts))
             except Exception as e:
-                fail(f"malformed entry in {path} (expected 'question_id' + 'options'): {e}")
+                fail(f"malformed entry in {path} (expected 'question_id' or 'data_index' + 'options'): {e}")
     if not qs:
         fail("no questions found in the release file(s).")
     return qs

requirements.txt

@@ -5,7 +5,6 @@
 # For GPU, install the CUDA build FIRST, then the rest, e.g.:
 #     pip install torch --index-url https://download.pytorch.org/whl/cu128
 #     pip install -r requirements.txt
-# (If a CPU-only torch is ALREADY installed, pip will skip the CUDA build — add --force-reinstall.)
 # The code auto-selects the GPU when a CUDA torch is present; CPU works too, just slower.
 torch>=2.2                # tested: 2.11.0+cu128 (CUDA); CPU wheel also works
 transformers>=4.40        # tested: 5.12.1  (microsoft/wavlm-large)