Paper-experiment prep: human-study harness + pass^k + paper plan

A coordinated batch landing the infrastructure the paper experiments
need before the full 200-pack run.

PAPER_PLAN.md — the working paper plan: central thesis, the
"perception not planning" framing, contributions, paper structure,
the three findings with sharpened claims + experiments + reviewer
attacks/defenses, §5 strategy-diversity (1v1 trajectory-embedding
figure), §12 pre-full-run audits (quality / coverage / pass^k),
limitations, stretch ideas.

Human-baseline study (the apples-to-apple human reference):
- openra_bench/human_study.py — frozen stratified 24-pack subset
(one pack per scenario family, 8 easy / 8 medium / 8 hard),
3 conditions (vision-fog / vision-clear / handoff-bad), a per-
player counterbalanced 72-cell playlist, and an
open_study_session() constructor that configures each condition
(vision-clear -> reveal_map; handoff-bad -> HANDOFF_K=3 stall
prefix auto-played).
- openra_bench/human_labeling.py — InteractiveSession.from_pack
gains a fog_mode param so the human Play path can run the no-fog
condition.
- app.py — Play-tab "Human-study mode" accordion: player id,
Begin study / Next scenario, progress tracker; walks a recruited
player through all 72 cells. Every game auto-saves apples-to-
apple with model runs.
- tests/test_human_study.py — subset packs verified to exist; the
three conditions verified live (vision-clear reveals 100% +
every enemy; handoff-bad starts HANDOFF_K turns deep); the
study session persists in the standard Playback format.

Multi-run reliability (the missing variance metric):
- openra_bench/run_eval.py — --repeats N runs each (cell, seed) N
times varying only model nondeterminism, so the report can carry
mean +- CI and pass^k (all-k wins) alongside pass@k. Each record
carries a repeat index; only rep=0 writes a Playback (no per-turn
dump bloat). --temperature T threads through ProviderConfig so
the repeats are meaningful.
- tests/test_run_eval_repeats.py — tasks multiply by N; records
carry the repeat index.

Full suite 8098 passed, 1 skipped, 3 xfailed, 2 xpassed.

PAPER_PLAN.md

@@ -0,0 +1,336 @@
+# OpenRA-Bench — Paper Plan & Critique
+
+Working document. Captures the central thesis, the paper framing, the
+three findings, the experiment program, and the open critiques /
+reviewer-defenses. Living doc — update as experiments land.
+
+---
+
+## 1. Central thesis (as stated)
+
+We build a benchmark on the Real-Time Strategy game *Red Alert* and
+measure LLM performance on **adversarial, multi-modal, long-horizon
+strategic planning and execution**. Three findings:
+
+1. **The major gap is image-based perception.** In pure-text
+   battlefields models execute complex strategies; in mixed modality
+   — where own and enemy positions must be read off a minimap — they
+   struggle. We show this by SFT on a small set of text-modality
+   results with the image attached, and observe improvement on this
+   benchmark *and* on external vision benchmarks (ERQA).
+2. **Models panic in unfavorable / uncertain situations.** Scores
+   drop sharply under fog of war while humans hold a baseline. Under
+   heavy losses, models choose `observe`/`stop` instead of actively
+   redirecting units. Hypothesis: a transfer of behavior from
+   code-heavy training, where the rewarded move in an unfavorable
+   state is to stop and wait for human intervention.
+3. **Bridging both gaps via simple SFT** lets a *smaller* model beat
+   large reasoning models on 1v1 ELO.
+
+Contributions: 200 research-grounded scenarios; an active 1v1
+battleground; evidence that models form complex, diverse strategies
+despite the perception and adversity gaps (and that strategy varies
+with temperature and is model-characteristic — turtle / balanced /
+aggressive).
+
+---
+
+## 2. Framing & positioning
+
+**The hook is the inversion:** LLMs can do the *hard* part
+(long-horizon adversarial strategy) but fail the "easy" parts
+(reading a map, holding their nerve). Lead with that — it inverts the
+usual "LLMs can't plan" narrative, and the failures are specific and
+*fixable*.
+
+Position OpenRA-Bench as a **diagnostic instrument**, not a
+leaderboard. The methodological novelty vs. the crowded
+SC2 / TextStarCraft LLM-agent space is the **ablation methodology that
+decomposes failure** into perception / reasoning / action / adversity.
+Arc of the paper: **diagnosis → localization → treatment → transfer.**
+
+Title candidates:
+- *Map-Blind and Panic-Prone: Diagnosing LLM Strategic Agents in
+  Real-Time Strategy*
+- *The Bottleneck Is Perception, Not Planning: A Diagnostic RTS
+  Benchmark*
+- *LLMs Can Strategize But Cannot See: Decomposing Agent Failure in
+  Red Alert*
+
+Venue: strong enough for a main track (ICLR/NeurIPS/ICML) on the
+diagnosis+treatment+transfer arc — not only Datasets & Benchmarks.
+
+---
+
+## 3. Contributions (sharpened)
+
+1. **OpenRA-Bench** — 200 controlled RTS scenarios on a deterministic
+   engine, each anchored to a named real-world capability / external
+   benchmark, built to a **no-cheat / no-defect** bar (every lazy /
+   brute / stall policy provably loses on every level and seed). The
+   validation rigor is itself a contribution: it is why the benchmark
+   measures the *intended* capability.
+2. **An ablation methodology that decomposes failure** — the
+   perception grid (channel × fog) and the handoff sweep — turning
+   "the model lost" into "the model lost *because of perception /
+   adversity / planning*."
+3. **A 1v1 self-play battleground** with ELO.
+4. **Two localized, fixable gaps** — perception-binding and
+   inaction-under-adversity — each with an SFT remedy, cross-benchmark
+   transfer (ERQA), and a small-beats-large result.
+
+---
+
+## 4. Paper structure
+
+§1 Intro (the inversion) · §2 Related work · §3 OpenRA-Bench (engine,
+200 scenarios, no-cheat design, ablation axes, 1v1/ELO,
+human-labeling) · §4 Setup (models, sweeps, metrics) · **§5 Models
+*can* strategize** (the diversity control) · §6 Finding 1: perception
+gap · §7 Finding 2: panic · §8 Finding 3: SFT remedy + transfer ·
+§9 Limitations · §10 Conclusion.
+
+**Key structural move:** promote the "diverse strategies" result from
+a bonus to **§5, before the failure findings.** Showing models produce
+coherent, diverse, model-characteristic strategies is what licenses
+the claim that the later failures are *not* failures of strategic
+reasoning. Without it a reviewer says "maybe they just can't plan."
+
+---
+
+## 5. §5 control — models *can* strategize
+
+Promote from "bonus" to a load-bearing control.
+
+- Classify each game's strategy (turtle / balanced / aggressive) via a
+  trajectory classifier (first-attack tick, army-vs-econ ratio,
+  expansion count, aggression index, build order). Show per-model
+  distributions differ.
+- Temperature sweep: strategy entropy vs. temperature for one model.
+- Show strategies are *coherent* (executed consistently once chosen),
+  not random.
+
+**The headline figure — the strategy-embedding scatter, in 1v1.** Each
+game's trajectory is featurized and embedded to 2D (PCA / UMAP);
+points colored by model. The plot shows, at a glance:
+- **inter-model clusters** — model A lives in the aggressive region,
+  model B turtles — models have characteristic strategy *priors*;
+- **intra-model spread** — the same model across games (especially at
+  high temperature) scatters — one model generates *diverse* play.
+1v1 is the right venue: full-macro adversarial games make strategy
+legible in a way single-scenario tasks do not. A bigger model roster
+(8–12, Together + Bedrock) makes the clustering visually striking.
+
+This is the evidence that planning is not the bottleneck — and a
+genuinely compelling standalone result.
+
+---
+
+## 6. Finding 1 — Perception gap
+
+**Claim, sharpened.** Decompose perception into **extraction** (read
+state from the image) and **binding** (act on image-derived state).
+The perception sweep already separates these: `image`-channel on
+perception packs (count / locate) isolates extraction; `image` on
+action packs isolates binding. Report *which* breaks. The result is
+"the perception→action **binding** fails while pure planning and pure
+extraction are intact" — not the weak "mixed modality is hard."
+
+**Experiments.**
+- Perception sweep (6 cells) × model roster × scenario set × seeds →
+  the modality-gap number per model (`structured − image`).
+- A "perceive-then-act" scaffold (force the model to transcribe
+  positions first) — if it rescues performance, the bottleneck is
+  binding, not extraction.
+- **Cross-modal action distillation SFT**: text-mode *winning*
+  trajectories + the attached minimap → train image→action. Eval
+  `image` channel on **held-out** scenarios.
+- **Transfer**: ERQA + ≥2 other spatial / vision benchmarks
+  (e.g. BLINK, a spatial-VQA set), before / after SFT.
+
+**Why transfer is the crown jewel.** Training image→action where the
+action is good could teach good *action priors* without the model
+ever using the image. The cross-benchmark transfer is what proves the
+SFT taught the model to *see*. Without it, finding 1 is "task
+finetuning helps task." With it, it is "agentic RTS data improves
+general visual spatial reasoning." Use ≥2 external benchmarks so it
+cannot be called cherry-picked.
+
+**Reviewer attack → defense.** "VLMs are known to be bad at dense
+small images — this is just OCR." → We measure it in a *consequential
+agentic loop*, we *localize* extraction vs binding, and we show a fix
+that *transfers*. Plus a render-style mini-ablation (the
+`constant_colors` / scale knobs) so the gap is not an artifact of one
+minimap style.
+
+---
+
+## 7. Finding 2 — Panic / inaction under adversity
+
+**Claim.** Models degrade far more than humans under fog; under heavy
+losses they default to `observe` / `stop` instead of active redirect.
+Operational definition = the **`passivity` metric** (fraction of the
+model's turns spent on `observe` / `stop` only). Keep "panic" only as
+an informal label; formal text says "inaction bias under adversity."
+
+**Experiments.**
+- Fog ablation × models × **humans** — report the *fog-penalty gap*
+  (human degrades little, model degrades a lot), not raw fog scores.
+- Handoff bad-prefix → passivity, models vs. human. Models freeze
+  (high passivity), humans redirect (low).
+- Show passivity is *causally costly* — within-model, passive turns
+  predict worse outcomes controlling for position quality.
+
+**The coding-bias hypothesis is the spiciest claim — handle with
+care.** Cannot be asserted as fact. Keep as a hypothesis AND support
+with ≥1 of:
+- **base vs. RLHF / instruct** versions of the same model — if base
+  models panic less, post-training is implicated;
+- a **loss-aware prompt intervention** ("inaction is costly; never
+  just observe when losing") — if a prompt largely fixes it, the
+  behavior is a shallow prior, consistent with the hypothesis;
+- passivity vs. known code / RLHF intensity across the roster.
+
+**Critical-path risk.** The human baseline needs *real human data at
+scale*. The Play tab is built; data is not collected. Scope: a
+representative scenario subset, several humans, disclosed RTS skill. A
+strong scripted policy can serve as a secondary "competent
+non-panicking" reference if human N is thin.
+
+---
+
+## 8. Finding 3 — SFT remedy, small beats large
+
+**Reviewer attack (serious).** "A task-finetuned small model beating a
+zero-shot large model is trivial." **Defenses to build in:**
+- The SFT is **small and targeted** (perception grounding + active
+  recovery), explicitly *not* trained on eval scenarios — state the
+  train / eval split loudly.
+- **Also SFT the large model** — show the gap is real and fixable at
+  every scale. Headline becomes: "the perception+panic gap is large
+  enough that closing it in a small model *outweighs a 10× reasoning
+  advantage* — and it was never a reasoning gap."
+- Ablate the SFT: perception-only / recovery-only / both — each
+  component contributes.
+- ELO with enough games + confidence intervals; held-out 1v1 maps.
+
+**Infrastructure synergy (state explicitly).** The handoff
+`TrajectoryController` + the human-Playback format *are* the SFT data
+pipeline — recovered bad-prefix episodes are active-recovery
+exemplars; text-mode wins are perception-distillation data. The
+ablation infrastructure doubles as the training-data factory.
+
+---
+
+## 9. Experiment program
+
+| # | Experiment | Infra | Status |
+|---|---|---|---|
+| §5 strategy diversity | classify 1v1 games; temp sweep; per-model dists | 1v1 harness ✓; needs strategy classifier | TODO |
+| Perception sweep | 6-cell × roster × scenarios × seeds | ✓ `--perception-sweep` | **run** |
+| Handoff / passivity | base/bad/good × roster | ✓ `--handoff-sweep` | **run** |
+| Human baseline | fog × scenario subset × humans | Play tab ✓, **no data** | **highest-risk** |
+| Cross-modal SFT + transfer | distill text→image; ERQA + 2 more | data pipeline ✓, needs finetuning | **biggest compute** |
+| 1v1 ELO tournament | round-robin + CIs | harness ✓ | run |
+| Recovery SFT | active-recovery exemplars → finetune | handoff bank ✓ | run |
+
+Critical path: human baseline (logistics) and the SFT (compute).
+
+---
+
+## 10. Metrics
+
+Win-rate / outcome · composite P/R/A score · objective-progress
+(continuous) · ELO (1v1, with CIs) · **passivity** (freeze metric) ·
+generalization gap (public vs. held-out seeds) · strategy class /
+entropy · human-normalized score (model / human) · derived gaps:
+modality gap = `score(structured) − score(image)`, fog penalty =
+`score(clear) − score(fog)`, per model and per human.
+
+---
+
+## 11. Threats to validity / limitations to preempt
+
+- **One game (RA).** Lean on the capability taxonomy
+  (`meta.benchmark_anchor`) + the ERQA transfer for generality.
+- **Engine is a reimplementation.** Deterministic + validated is the
+  answer.
+- **One minimap render style.** Render-robustness ablation.
+- **Human skill / N.** Disclose; representative subset.
+- **"Panic = code training."** Hypothesis, not claim — support with
+  the probes in §7.
+- **SFT leakage.** Loud train/eval scenario split.
+- **ELO methodology.** Game count, pairing, confidence intervals.
+
+---
+
+## 12. Pre-full-run audits (must land before the 200-pack sweep)
+
+After the pilot finishes and *before* committing compute to the full
+200-pack run, three audits gate the rigor of the headline numbers:
+
+### 12.1 Scenario quality audit
+Two layers:
+- **Static** — re-run the scripted-policy bar (`stall` / `brute` /
+  `intended`) across all 200 packs. Engine fixes may have drifted a
+  pack since authoring (a lazy policy now wins, or `intended` now
+  loses). Catches benchmark rot.
+- **Empirical** — from pilot/full-run data, flag packs where *every*
+  model wins (too easy / a trivial idiom dominates — task #43) or
+  *every* model loses (unsolvable or a predicate is mis-tuned —
+  task #44). Discriminative packs are the only useful ones.
+
+Paper payoff: a post-hoc audit table converts the "no-defect bar"
+claim into something you can *show* — a strong methodology subsection.
+
+### 12.2 Coverage map — RTS phase × decision-divergence
+Map all 200 packs (plus the 1v1 battleground) onto the **RTS phase ×
+decision-divergence matrix** from the original plan
+(opening / early-mid / mid / mid-late / late × the canonical decisions
+in each). Produce a coverage heatmap; flag empty / thin cells. Surface
+the `meta.capability`-tag imbalance (`adversarial`=1 pack — full
+end-to-end macro lives in the 1v1 battleground, both belong on the
+map). Paper payoff: a figure showing the bench spans the *real* game,
+not just easy probes.
+
+### 12.3 Multi-run reliability — `pass^k`
+Each (cell, seed) is run N times varying only model nondeterminism
+(requires temperature > 0). Report mean ± CI **and** `pass^k`
+(all-k-wins). A model that wins 5/10 identical runs is a fundamentally
+different finding than 10/10. `--repeats N` in `run_eval`; default
+`k=5` (Codex / SWE-bench convention). Paper payoff: mean-only is
+fragile; reliability is itself a possible headline result.
+
+## 13. Stretch ideas
+
+- **Pivotal-turn analysis** — single-turn counterfactual swaps to show
+  RTS losses are 1–2 catastrophic decisions, not uniform decay.
+
+---
+
+## 13. Ablation infrastructure already built (this is real, today)
+
+- **Fog axis** — engine `reveal_map` no-fog flag (`OpenRA-Rust`),
+  the `-clear` perception cells.
+- **Modality axis** — `structured` / `vision` / `image` (image-
+  primary, text redacted, labelled minimap) channels;
+  `run_eval --perception-sweep` expands `pack:level` into the 6
+  modality cells.
+- **Handoff axis** — `openra_bench/handoff.py`
+  (`HandoffController`, `TrajectoryController`), `run_eval
+  --handoff-sweep`; the `passivity` metric on every result.
+- **1v1 battleground + ELO** — `one_v_one.py`, scripted ladder.
+- **Human-labeling** — the Play tab persists human runs in the
+  standard `Playback` format (apples-to-apples with model runs).
+- **200 scenario packs** — no-cheat-validated, capability-anchored.
+
+---
+
+## 14. Open decisions
+
+- Model roster for the sweeps (which models; vision-capable required
+  for `vision` / `image` channels).
+- Compute / API budget for the full sweeps.
+- Human-study scope (how many humans, which scenario subset).
+- SFT base model(s) and the small/large pairing for finding 3.
+- Strategy-classifier definition for §5.

app.py

@@ -1242,6 +1242,76 @@ def _play_start(prev_sess, pack, level, seed):
     )
 
 
+# ── Human-study mode ─────────────────────────────────────────────────
+# Walks a recruited player through the fixed 24-pack study subset under
+# 3 conditions (72 games, per-player counterbalanced). Every game saves
+# to the standard Playback format — apples-to-apple with model runs.
+
+def _study_progress_md(st: dict) -> str:
+    pl = st.get("playlist", [])
+    i = st.get("idx", 0)
+    if not pl:
+        return "_Enter your name and click **Begin study**._"
+    if i >= len(pl):
+        return (
+            f"**✅ Study complete** — all {len(pl)} games done. "
+            f"Thank you, `{st['player']}`!"
+        )
+    pack, level, cond = pl[i]
+    return (
+        f"**Study — game {i + 1} / {len(pl)}**  ·  player `{st['player']}`\n\n"
+        f"`{pack}` [{level}]  ·  condition: **{cond}**  \n"
+        f"_Play to game-over, then click **Next scenario ▶**._"
+    )
+
+
+def _study_render(st: dict, prev_sess):
+    """Open the study session for st's current cell and render it."""
+    if prev_sess is not None:
+        try:
+            prev_sess.close()
+        except Exception:  # noqa: BLE001
+            pass
+    empty = _play_units_df(None, [])
+    pl = st.get("playlist", [])
+    if st.get("idx", 0) >= len(pl):
+        return (None, [], [], "", None, "", "", empty, st,
+                _study_progress_md(st))
+    pack, level, cond = pl[st["idx"]]
+    try:
+        from openra_bench.human_study import open_study_session
+
+        sess = open_study_session(
+            pack, level, cond, player=st["player"], seed=1
+        )
+    except Exception as e:  # noqa: BLE001
+        return (None, [], [], "", None, f"⚠️ {e}",
+                "_study load failed_", empty, st, _study_progress_md(st))
+    img, brief, status, units = _play_render(sess, [], [])
+    return (sess, [], [], _play_objective_md(sess), img, brief, status,
+            units, st, _study_progress_md(st))
+
+
+def _study_begin(prev_sess, player):
+    import hashlib
+
+    from openra_bench.human_study import study_playlist
+
+    player = (player or "").strip() or "anon"
+    # Per-player counterbalancing — a stable seed from the name.
+    seed = int(hashlib.md5(player.encode()).hexdigest()[:8], 16)
+    st = {"player": player, "playlist": study_playlist(seed), "idx": 0}
+    return _study_render(st, prev_sess)
+
+
+def _study_next(prev_sess, st):
+    if not st or "playlist" not in st:
+        return _study_render({"player": "anon", "playlist": []}, prev_sess)
+    st = dict(st)
+    st["idx"] = st.get("idx", 0) + 1
+    return _study_render(st, prev_sess)
+
+
 def _play_click(sess, sel, queue, evt: gr.SelectData):
     """Contextual minimap click — classic RTS interaction, no mode:
 
@@ -1567,6 +1637,29 @@ def build_app() -> gr.Blocks:
                 play_sess = gr.State(None)
                 play_sel = gr.State([])
                 play_queue = gr.State([])
+                study_state = gr.State({})
+                with gr.Accordion(
+                    "📋 Human-study mode — 24-pack subset, 3 conditions",
+                    open=False,
+                ):
+                    gr.Markdown(
+                        "For the **human-baseline study**. Enter your name "
+                        "and click **Begin study** — you'll be walked "
+                        "through 72 games (24 scenarios × fog / no-fog / "
+                        "handoff-deficit), counterbalanced per player. "
+                        "Play each to game-over, then **Next scenario ▶**. "
+                        "Every game auto-saves apples-to-apple with the "
+                        "model runs."
+                    )
+                    with gr.Row():
+                        study_player = gr.Textbox(
+                            label="Your name / id", scale=2,
+                        )
+                        study_begin_btn = gr.Button("Begin study", scale=1)
+                        study_next_btn = gr.Button(
+                            "Next scenario ▶", variant="primary", scale=1,
+                        )
+                    study_progress = gr.Markdown()
                 with gr.Row():
                     play_scen = gr.Dropdown(
                         choices=_play_scenarios(), label="Scenario",
@@ -1613,6 +1706,19 @@ def build_app() -> gr.Blocks:
                         play_img, play_brief, play_status, play_units,
                     ],
                 )
+                _study_outputs = [
+                    play_sess, play_sel, play_queue, play_objective,
+                    play_img, play_brief, play_status, play_units,
+                    study_state, study_progress,
+                ]
+                study_begin_btn.click(
+                    _study_begin, inputs=[play_sess, study_player],
+                    outputs=_study_outputs,
+                )
+                study_next_btn.click(
+                    _study_next, inputs=[play_sess, study_state],
+                    outputs=_study_outputs,
+                )
                 play_img.select(
                     _play_click,
                     inputs=[play_sess, play_sel, play_queue],

openra_bench/human_labeling.py

@@ -535,14 +535,19 @@ class InteractiveSession:
         record: bool = True,
         playback_root: Any = None,
         player: str = "Human",
+        fog_mode: str = "vision",
     ) -> "InteractiveSession":
-        """Compile a pack by id and open a session on it."""
+        """Compile a pack by id and open a session on it. `fog_mode`
+        selects the observation modality — `vision-clear` (or any
+        `-clear` mode) reveals the whole map (engine `reveal_map`), the
+        no-fog condition of the human-study fog-penalty pair."""
         from .scenarios import load_pack
         from .scenarios.loader import PACKS_DIR, compile_level
 
         compiled = compile_level(
             load_pack(PACKS_DIR / f"{pack_id}.yaml"), level
         )
+        compiled.fog_mode = fog_mode
         return cls(
             compiled, seed=seed, record=record,
             playback_root=playback_root, player=player,

openra_bench/human_study.py

@@ -0,0 +1,120 @@
+"""Human-baseline study — the apples-to-apple human reference.
+
+External tournament replays cannot be scored on this benchmark: a real
+RTS game is a different engine, continuous-time, a different action
+space, and not one of the constructed tasks. The ONLY comparable human
+data comes through the Play tab — same engine, scenario, observation,
+tool API, win predicate, and `Playback` format the models use.
+
+This module defines a fixed, stratified 24-pack subset (one pack per
+scenario family, difficulty spread 8 easy / 8 medium / 8 hard) and the
+three conditions a player plays each under:
+
+* `vision-fog`   — the canonical fogged minimap (normal play).
+* `vision-clear` — no fog (engine `reveal_map`): the fog-penalty pair.
+* `handoff-bad`  — the player inherits a losing position (a `stall`
+                   prefix of `HANDOFF_K` turns), then plays on — the
+                   recovery / freeze comparison.
+
+A per-player counterbalanced playlist walks all 24 x 3 = 72 cells.
+"""
+
+from __future__ import annotations
+
+import random
+from typing import Any
+
+# Frozen subset — one representative pack per scenario family, levels
+# spread across difficulty. Stratified by family (not by the lopsided
+# `meta.capability` tag). Keep this list STABLE — it pins the baseline.
+STUDY_SUBSET: list[tuple[str, str]] = [
+    ("combat-pincer-coordination", "easy"),
+    ("econ-overflow-to-silos", "medium"),
+    ("def-pre-position-mobile-reserve", "hard"),
+    ("build-rally-point-management", "easy"),
+    ("scout-detect-incoming-army", "medium"),
+    ("lh-opening-to-defense-to-counter", "hard"),
+    ("proc-only-defend-no-attack", "easy"),
+    ("mfb-supply-line-link-between-bases", "medium"),
+    ("rob-objective-shift-with-or-clause", "hard"),
+    ("coord-mutual-support", "easy"),
+    ("tp-rush-objective-very-fast", "medium"),
+    ("mcv-deploy-relocate-under-pressure", "hard"),
+    ("artofwar-lure-the-tiger", "easy"),
+    ("economy-harvest-timebox", "medium"),
+    ("perception-frontier-reading", "hard"),
+    ("strategy-trilemma", "easy"),
+    ("tech-production-planning", "medium"),
+    ("expansion-balanced-2-base-defended", "hard"),
+    ("mid-economy-under-fire", "easy"),
+    ("strict-sequence", "medium"),
+    ("action-sequenced-execution", "hard"),
+    ("adv-rps-counter-pick", "easy"),
+    ("coordination-staggered-window", "medium"),
+    ("maint-sell-and-recoup-cash", "hard"),
+]
+
+STUDY_CONDITIONS: tuple[str, ...] = ("vision-fog", "vision-clear", "handoff-bad")
+
+# Prefix length for the `handoff-bad` condition — the player inherits a
+# game already `HANDOFF_K` observe-only (stall) turns deep.
+HANDOFF_K = 3
+
+# fog_mode each condition compiles the scenario under.
+_CONDITION_FOG = {
+    "vision-fog": "vision",
+    "vision-clear": "vision-clear",
+    "handoff-bad": "vision",
+}
+
+
+def study_playlist(player_seed: int = 0) -> list[tuple[str, str, str]]:
+    """The 72-cell playlist — every (pack, level) x condition — in a
+    per-player counterbalanced (deterministically shuffled) order, so
+    condition ordering is not confounded across players."""
+    cells = [
+        (pack, level, cond)
+        for pack, level in STUDY_SUBSET
+        for cond in STUDY_CONDITIONS
+    ]
+    random.Random(player_seed).shuffle(cells)
+    return cells
+
+
+def open_study_session(
+    pack: str,
+    level: str,
+    condition: str,
+    player: str,
+    seed: int = 1,
+    playback_root: Any = None,
+):
+    """Open an `InteractiveSession` for one study cell, configured for
+    `condition`. For `handoff-bad` the engine is advanced `HANDOFF_K`
+    observe-only turns BEFORE the player takes over — so the player
+    inherits a real deficit, exactly like the model handoff ablation.
+
+    The run persists to the standard `Playback` format (apples-to-apple
+    with model runs); `playback_root` defaults to a per-condition dir
+    so the condition is recoverable from the path."""
+    from pathlib import Path
+
+    from .human_labeling import InteractiveSession
+
+    if condition not in STUDY_CONDITIONS:
+        raise ValueError(f"unknown study condition {condition!r}")
+    if playback_root is None:
+        playback_root = Path("playback/human_study") / condition
+
+    sess = InteractiveSession.from_pack(
+        pack, level, seed,
+        record=True, playback_root=playback_root, player=player,
+        fog_mode=_CONDITION_FOG[condition],
+    )
+    if condition == "handoff-bad":
+        # Stall prefix — the player inherits the resulting losing board.
+        for _ in range(HANDOFF_K):
+            if sess.done:
+                break
+            sess.submit_turn([])
+    return sess

openra_bench/run_eval.py

@@ -165,6 +165,7 @@ def evaluate(
     handoff_sweep: bool = False,
     handoff_k: int = 3,
     handoff_bank: str | Path | None = None,
+    repeats: int = 1,
 ) -> dict:
     """Run packs×levels×seeds. If `held_out_seeds` is given, those are
     run too and tagged split='held_out'; the report adds
@@ -184,6 +185,11 @@ def evaluate(
     position replayed from a `handoff_bank` trajectory (`good` — the
     capitalize-on-advantage test). `handoff_k` is the prefix length.
     Each record carries a `passivity` stat (observe/stop-only fraction).
+
+    `repeats` runs each (cell, seed) `N` times, varying only model
+    nondeterminism (assumes temperature > 0). Records carry a `repeat`
+    index 0..N-1, so aggregation can report mean ± CI and `pass^k`
+    (all-k wins) alongside `pass@k` — the reliability metric.
     """
     from .resilience import (
         BudgetExceeded,
@@ -309,12 +315,16 @@ def evaluate(
                 continue
             for split, slist in (("public", seeds), ("held_out", held_out_seeds)):
                 for seed in slist:
-                    tasks.append((compiled, cell, split, seed))
+                    for rep in range(max(1, repeats)):
+                        tasks.append((compiled, cell, split, seed, rep))
 
     def _run_one(task: tuple) -> dict:
-        compiled, cell, split, seed = task
+        compiled, cell, split, seed, rep = task
         pb = None
-        if playback_root is not None:
+        # Only the first repeat writes a Playback — the records (the
+        # lightweight per-rep results) carry the pass^k data; saving N
+        # full per-turn dumps per cell would just bloat disk.
+        if playback_root is not None and rep == 0:
             from .playback import Playback
 
             pb = Playback(
@@ -359,6 +369,7 @@ def evaluate(
             "capability": compiled.meta.capability,
             "split": split,
             "seed": seed,
+            "repeat": rep,
             "outcome": sc.outcome,
             "composite": sc.composite,
             "perception": sc.perception,
@@ -446,7 +457,7 @@ def evaluate(
             # not abort a multi-hour sweep or lose the report — record
             # it as outcome="error" and continue. Budget is the only
             # signal that intentionally stops the whole run.
-            compiled, cell, split, seed = task
+            compiled, cell, split, seed, rep = task
             try:
                 return _run_one(task)
             except BudgetExceeded:
@@ -458,6 +469,7 @@ def evaluate(
                     "capability": compiled.meta.capability,
                     "split": split,
                     "seed": seed,
+                    "repeat": rep,
                     "outcome": "error",
                     "composite": 0.0,
                     "perception": 0.0,
@@ -688,6 +700,14 @@ def main(argv: list[str]) -> int:
     ap.add_argument("--handoff-bank", default=None,
                     help="dir of Playback runs — source of winning "
                     "trajectories for the handoff-good prefix")
+    ap.add_argument("--repeats", type=int, default=1,
+                    help="run each (cell, seed) N times varying only "
+                    "model nondeterminism — enables mean +- CI and "
+                    "pass^k reliability metrics (needs temperature > 0)")
+    ap.add_argument("--temperature", type=float, default=None,
+                    help="sampling temperature for the model "
+                    "(overrides ProviderConfig.temperature). Set > 0 "
+                    "to make --repeats meaningful.")
     a = ap.parse_args(argv[1:])
 
     cfg = None
@@ -704,7 +724,7 @@ def main(argv: list[str]) -> int:
             if quant:
                 pr["quantizations"] = [quant]
             extra_body["provider"] = pr
-        cfg = ProviderConfig(
+        cfg_kw = dict(
             provider=a.provider,
             model=a.model,
             base_url=a.base_url,
@@ -713,6 +733,9 @@ def main(argv: list[str]) -> int:
             fog_mode=a.fog_mode,
             extra_body=extra_body,
         )
+        if a.temperature is not None:
+            cfg_kw["temperature"] = a.temperature
+        cfg = ProviderConfig(**cfg_kw)
 
     stats = evaluate(
         _resolve_packs(a.packs),
@@ -733,6 +756,7 @@ def main(argv: list[str]) -> int:
         handoff_sweep=a.handoff_sweep,
         handoff_k=a.handoff_k,
         handoff_bank=a.handoff_bank,
+        repeats=a.repeats,
         progress=lambda d, n, rec, c: print(
             f"[{d}/{n}] {rec['cell']}:{rec['split']}#{rec['seed']} "
             f"{rec['outcome']} comp={rec['composite']} "

tests/test_human_study.py

@@ -0,0 +1,78 @@
+"""The human-baseline study harness — the apples-to-apple human
+reference: a fixed 24-pack subset, three conditions, a per-player
+counterbalanced playlist, run through the same Play-tab `Playback`
+pipeline the models use."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
+
+from openra_bench.human_study import (HANDOFF_K, STUDY_CONDITIONS,
+                                      STUDY_SUBSET, open_study_session,
+                                      study_playlist)
+
+PACKS = Path(__file__).parent.parent / "openra_bench" / "scenarios" / "packs"
+
+
+def test_subset_is_24_packs_that_all_exist():
+    assert len(STUDY_SUBSET) == 24
+    for pack, level in STUDY_SUBSET:
+        assert (PACKS / f"{pack}.yaml").exists(), pack
+        assert level in ("easy", "medium", "hard"), (pack, level)
+    # difficulty is stratified, not all one level
+    levels = {lvl for _, lvl in STUDY_SUBSET}
+    assert levels == {"easy", "medium", "hard"}
+
+
+def test_playlist_is_full_grid_and_counterbalanced():
+    pl0 = study_playlist(player_seed=0)
+    assert len(pl0) == len(STUDY_SUBSET) * len(STUDY_CONDITIONS) == 72
+    # every (pack, level, condition) cell appears exactly once
+    assert len(set(pl0)) == 72
+    for pack, level, cond in pl0:
+        assert (pack, level) in STUDY_SUBSET
+        assert cond in STUDY_CONDITIONS
+    # different players get a different order (counterbalancing)
+    assert study_playlist(1) != pl0
+
+
+@pytest.mark.parametrize("cond", STUDY_CONDITIONS)
+def test_open_study_session_each_condition(cond, tmp_path):
+    sess = open_study_session(
+        "perception-frontier-reading", "hard", cond,
+        player="tester", seed=1, playback_root=tmp_path,
+    )
+    try:
+        rs = sess.render_state()
+        if cond == "vision-clear":
+            # no fog — engine reveal_map: whole map explored
+            assert rs.get("explored_percent", 0) > 99.0
+            assert sess.compiled.fog_mode == "vision-clear"
+        elif cond == "handoff-bad":
+            # the player inherits a HANDOFF_K-turn-deep deficit
+            assert sess.turn == HANDOFF_K
+            assert sess.compiled.fog_mode == "vision"
+        else:  # vision-fog — normal fogged start
+            assert sess.turn == 0
+            assert rs.get("explored_percent", 100) < 50.0
+    finally:
+        sess.close()
+
+
+def test_study_session_persists_playback(tmp_path):
+    """A study game must save in the standard Playback format so it is
+    apples-to-apple with model runs."""
+    sess = open_study_session(
+        "perception-frontier-reading", "easy", "vision-fog",
+        player="tester", seed=1, playback_root=tmp_path,
+    )
+    try:
+        sess.submit_turn([])  # one observe turn
+        assert sess._playback is not None
+        assert sess._playback.dir.exists()
+    finally:
+        sess.close()

tests/test_run_eval_repeats.py

@@ -0,0 +1,35 @@
+"""--repeats N: run each (cell, seed) N times so the report carries
+the variance + pass^k a single inference per cell cannot give."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
+
+from openra_bench.run_eval import evaluate
+
+PACK = (
+    Path(__file__).parent.parent
+    / "openra_bench" / "scenarios" / "packs"
+    / "perception-count-the-threat.yaml"
+)
+
+
+def test_repeats_multiplies_the_task_count():
+    base = evaluate([PACK], levels=["easy"], seeds=[1, 2],
+                    repeats=1, dry_run=True)
+    rep3 = evaluate([PACK], levels=["easy"], seeds=[1, 2],
+                    repeats=3, dry_run=True)
+    assert rep3["tasks"] == base["tasks"] * 3
+
+
+def test_records_carry_repeat_index():
+    stats = evaluate([PACK], levels=["easy"], seeds=[1], repeats=3)
+    eps = stats.get("episodes", [])
+    assert len(eps) == 3
+    assert {e["repeat"] for e in eps} == {0, 1, 2}
+    # all repeats stay at the same (cell, seed) — same key, different rep
+    assert len({(e["cell"], e["seed"]) for e in eps}) == 1