PAPER_PLAN.md

# OpenRA-Bench — Paper Plan & Critique

Working document. Captures the central thesis, the paper framing, the
three findings, the experiment program, and the open critiques /
reviewer-defenses. Living doc — update as experiments land.

---

## 1. Central thesis (as stated)

We build a benchmark on the Real-Time Strategy game *Red Alert* and
measure LLM performance on **adversarial, multi-modal, long-horizon
strategic planning and execution**. Three findings:

1. **The major gap is image-based perception.** In pure-text
   battlefields models execute complex strategies; in mixed modality
   — where own and enemy positions must be read off a minimap — they
   struggle. We show this by SFT on a small set of text-modality
   results with the image attached, and observe improvement on this
   benchmark *and* on external vision benchmarks (ERQA).
2. **Models panic in unfavorable / uncertain situations.** Scores
   drop sharply under fog of war while humans hold a baseline. Under
   heavy losses, models choose `observe`/`stop` instead of actively
   redirecting units. Hypothesis: a transfer of behavior from
   code-heavy training, where the rewarded move in an unfavorable
   state is to stop and wait for human intervention.
3. **Bridging both gaps via simple SFT** lets a *smaller* model beat
   large reasoning models on 1v1 ELO.

Contributions: 200 research-grounded scenarios; an active 1v1
battleground; evidence that models form complex, diverse strategies
despite the perception and adversity gaps (and that strategy varies
with temperature and is model-characteristic — turtle / balanced /
aggressive).

---

## 2. Framing & positioning

**The hook is the inversion:** LLMs can do the *hard* part
(long-horizon adversarial strategy) but fail the "easy" parts
(reading a map, holding their nerve). Lead with that — it inverts the
usual "LLMs can't plan" narrative, and the failures are specific and
*fixable*.

Position OpenRA-Bench as a **diagnostic instrument**, not a
leaderboard. The methodological novelty vs. the crowded
SC2 / TextStarCraft LLM-agent space is the **ablation methodology that
decomposes failure** into perception / reasoning / action / adversity.
Arc of the paper: **diagnosis → localization → treatment → transfer.**

Title candidates:
- *Map-Blind and Panic-Prone: Diagnosing LLM Strategic Agents in
  Real-Time Strategy*
- *The Bottleneck Is Perception, Not Planning: A Diagnostic RTS
  Benchmark*
- *LLMs Can Strategize But Cannot See: Decomposing Agent Failure in
  Red Alert*

Venue: strong enough for a main track (ICLR/NeurIPS/ICML) on the
diagnosis+treatment+transfer arc — not only Datasets & Benchmarks.

---

## 3. Contributions (sharpened)

1. **OpenRA-Bench** — 200 controlled RTS scenarios on a deterministic
   engine, each anchored to a named real-world capability / external
   benchmark, built to a **no-cheat / no-defect** bar (every lazy /
   brute / stall policy provably loses on every level and seed). The
   validation rigor is itself a contribution: it is why the benchmark
   measures the *intended* capability.
2. **An ablation methodology that decomposes failure** — the
   perception grid (channel × fog) and the handoff sweep — turning
   "the model lost" into "the model lost *because of perception /
   adversity / planning*."
3. **A 1v1 self-play battleground** with ELO.
4. **Two localized, fixable gaps** — perception-binding and
   inaction-under-adversity — each with an SFT remedy, cross-benchmark
   transfer (ERQA), and a small-beats-large result.

---

## 4. Paper structure

§1 Intro (the inversion) · §2 Related work · §3 OpenRA-Bench (engine,
200 scenarios, no-cheat design, ablation axes, 1v1/ELO,
human-labeling) · §4 Setup (models, sweeps, metrics) · **§5 Models
*can* strategize** (the diversity control) · §6 Finding 1: perception
gap · §7 Finding 2: panic · §8 Finding 3: SFT remedy + transfer ·
§9 Limitations · §10 Conclusion.

**Key structural move:** promote the "diverse strategies" result from
a bonus to **§5, before the failure findings.** Showing models produce
coherent, diverse, model-characteristic strategies is what licenses
the claim that the later failures are *not* failures of strategic
reasoning. Without it a reviewer says "maybe they just can't plan."

---

## 5. §5 control — models *can* strategize

Promote from "bonus" to a load-bearing control.

- Classify each game's strategy (turtle / balanced / aggressive) via a
  trajectory classifier (first-attack tick, army-vs-econ ratio,
  expansion count, aggression index, build order). Show per-model
  distributions differ.
- Temperature sweep: strategy entropy vs. temperature for one model.
- Show strategies are *coherent* (executed consistently once chosen),
  not random.

**The headline figure — the strategy-embedding scatter, in 1v1.** Each
game's trajectory is featurized and embedded to 2D (PCA / UMAP);
points colored by model. The plot shows, at a glance:
- **inter-model clusters** — model A lives in the aggressive region,
  model B turtles — models have characteristic strategy *priors*;
- **intra-model spread** — the same model across games (especially at
  high temperature) scatters — one model generates *diverse* play.
1v1 is the right venue: full-macro adversarial games make strategy
legible in a way single-scenario tasks do not. A bigger model roster
(8–12, Together + Bedrock) makes the clustering visually striking.

This is the evidence that planning is not the bottleneck — and a
genuinely compelling standalone result.

---

## 6. Finding 1 — Perception gap

**Claim, sharpened.** Decompose perception into **extraction** (read
state from the image) and **binding** (act on image-derived state).
The perception sweep already separates these: `image`-channel on
perception packs (count / locate) isolates extraction; `image` on
action packs isolates binding. Report *which* breaks. The result is
"the perception→action **binding** fails while pure planning and pure
extraction are intact" — not the weak "mixed modality is hard."

**Experiments.**
- Perception sweep (6 cells) × model roster × scenario set × seeds →
  the modality-gap number per model (`structured − image`).
- A "perceive-then-act" scaffold (force the model to transcribe
  positions first) — if it rescues performance, the bottleneck is
  binding, not extraction.
- **Cross-modal action distillation SFT**: text-mode *winning*
  trajectories + the attached minimap → train image→action. Eval
  `image` channel on **held-out** scenarios.
- **Transfer**: ERQA + ≥2 other spatial / vision benchmarks
  (e.g. BLINK, a spatial-VQA set), before / after SFT.

**Why transfer is the crown jewel.** Training image→action where the
action is good could teach good *action priors* without the model
ever using the image. The cross-benchmark transfer is what proves the
SFT taught the model to *see*. Without it, finding 1 is "task
finetuning helps task." With it, it is "agentic RTS data improves
general visual spatial reasoning." Use ≥2 external benchmarks so it
cannot be called cherry-picked.

**Reviewer attack → defense.** "VLMs are known to be bad at dense
small images — this is just OCR." → We measure it in a *consequential
agentic loop*, we *localize* extraction vs binding, and we show a fix
that *transfers*. Plus a render-style mini-ablation (the
`constant_colors` / scale knobs) so the gap is not an artifact of one
minimap style.

---

## 7. Finding 2 — Panic / inaction under adversity

**Claim.** Models degrade far more than humans under fog; under heavy
losses they default to `observe` / `stop` instead of active redirect.
Operational definition = the **`passivity` metric** (fraction of the
model's turns spent on `observe` / `stop` only). Keep "panic" only as
an informal label; formal text says "inaction bias under adversity."

**Experiments.**
- Fog ablation × models × **humans** — report the *fog-penalty gap*
  (human degrades little, model degrades a lot), not raw fog scores.
- Handoff bad-prefix → passivity, models vs. human. Models freeze
  (high passivity), humans redirect (low).
- Show passivity is *causally costly* — within-model, passive turns
  predict worse outcomes controlling for position quality.

**The coding-bias hypothesis is the spiciest claim — handle with
care.** Cannot be asserted as fact. Keep as a hypothesis AND support
with ≥1 of:
- **base vs. RLHF / instruct** versions of the same model — if base
  models panic less, post-training is implicated;
- a **loss-aware prompt intervention** ("inaction is costly; never
  just observe when losing") — if a prompt largely fixes it, the
  behavior is a shallow prior, consistent with the hypothesis;
- passivity vs. known code / RLHF intensity across the roster.

**Critical-path risk.** The human baseline needs *real human data at
scale*. The Play tab is built; data is not collected. Scope: a
representative scenario subset, several humans, disclosed RTS skill. A
strong scripted policy can serve as a secondary "competent
non-panicking" reference if human N is thin.

---

## 8. Finding 3 — SFT remedy, small beats large

**Reviewer attack (serious).** "A task-finetuned small model beating a
zero-shot large model is trivial." **Defenses to build in:**
- The SFT is **small and targeted** (perception grounding + active
  recovery), explicitly *not* trained on eval scenarios — state the
  train / eval split loudly.
- **Also SFT the large model** — show the gap is real and fixable at
  every scale. Headline becomes: "the perception+panic gap is large
  enough that closing it in a small model *outweighs a 10× reasoning
  advantage* — and it was never a reasoning gap."
- Ablate the SFT: perception-only / recovery-only / both — each
  component contributes.
- ELO with enough games + confidence intervals; held-out 1v1 maps.

**Infrastructure synergy (state explicitly).** The handoff
`TrajectoryController` + the human-Playback format *are* the SFT data
pipeline — recovered bad-prefix episodes are active-recovery
exemplars; text-mode wins are perception-distillation data. The
ablation infrastructure doubles as the training-data factory.

---

## 9. Experiment program

| # | Experiment | Infra | Status |
|---|---|---|---|
| §5 strategy diversity | classify 1v1 games; temp sweep; per-model dists | 1v1 harness ✓; needs strategy classifier | TODO |
| Perception sweep | 6-cell × roster × scenarios × seeds | ✓ `--perception-sweep` | **run** |
| Handoff / passivity | base/bad/good × roster | ✓ `--handoff-sweep` | **run** |
| Human baseline | fog × scenario subset × humans | Play tab ✓, **no data** | **highest-risk** |
| Cross-modal SFT + transfer | distill text→image; ERQA + 2 more | data pipeline ✓, needs finetuning | **biggest compute** |
| 1v1 ELO tournament | round-robin + CIs | harness ✓ | run |
| Recovery SFT | active-recovery exemplars → finetune | handoff bank ✓ | run |

Critical path: human baseline (logistics) and the SFT (compute).

---

## 10. Metrics

Win-rate / outcome · composite P/R/A score · objective-progress
(continuous) · ELO (1v1, with CIs) · **passivity** (freeze metric) ·
generalization gap (public vs. held-out seeds) · strategy class /
entropy · human-normalized score (model / human) · derived gaps:
modality gap = `score(structured) − score(image)`, fog penalty =
`score(clear) − score(fog)`, per model and per human.

---

## 11. Threats to validity / limitations to preempt

### 11.1 Out-of-scope engine features (paper must scope around them)
The Rust engine is a *RA-Lite* — ground-only, no resource layer.
The following features are **not implemented** and the bench has
zero packs for them. They are documented future work, NOT silently
missing:

- **Engineer capture** (`capture_actor`) — task #11 (S8).
- **Superweapons** — nuke, iron-curtain, chronosphere — S8.
- **Spies / thief** — infiltration, steal — S8.
- **Tanya** (Allied commando hero unit) — new unit type, not in plan.
- **Air units** — yak / mig / heli — needs `Aircraft` trait + flight.
- **Naval** — dd / ca / pt / lst + water mapgen.
- **Resource layer / ore patches** — `Resource` trait + harvester
  contention. The 1v1 map `rush-hour-arena` has **no ore patches**;
  economy is driven by `starting_cash` only. No mining contestation.
- **APC ground transport** — engine HAS `enter_transport` /
  `unload` + cargo storage; the bench has ~1 pack — could author
  more but the mechanism is sound.

**Paper scope:** "macro economy + combat micro + multi-base +
perception, in a ground-only RA-Lite engine." The features above are
documented as out-of-scope; reviewers will see the explicit list.

### 11.2 Methodological caveats (the standard list)
- **One game (RA).** Lean on the capability taxonomy
  (`meta.benchmark_anchor`) + the ERQA transfer for generality.
- **Engine is a reimplementation.** Deterministic + validated is the
  answer.
- **One minimap render style.** Render-robustness ablation.
- **Human skill / N.** Disclose; representative subset.
- **"Panic = code training."** Hypothesis, not claim — support with
  the probes in §7.
- **SFT leakage.** Loud train/eval scenario split.
- **ELO methodology.** Game count, pairing, confidence intervals.

### 11.3 Triage coverage (`scripts/triage.py`)
Per-pack `INTENDED` policy attestation comes from each pack's
dedicated `tests/test_<pack>.py` file (when present) — every such
test is in the suite and the suite is green, so the test passing
proves the intended policy still wins against the current engine.
Post defect-fix wave:
- **167 / 196 packs** (85%) have a dedicated test → "VERIFIED."
- **29 / 196 packs** (15%) are stall-bar-only verified (no test).
  Either add a test or rely on full-run empirical attestation.
- **1 pack** (`def-with-ambush`) is exempt by design (positional-
  discipline scenario where do-nothing IS the intended policy).
- **0 packs** fail the stall-must-lose bar.

---

## 12. Pre-full-run audits (must land before the 200-pack sweep)

After the pilot finishes and *before* committing compute to the full
200-pack run, three audits gate the rigor of the headline numbers:

### 12.1 Scenario quality audit
Two layers:
- **Static** — re-run the scripted-policy bar (`stall` / `brute` /
  `intended`) across all 200 packs. Engine fixes may have drifted a
  pack since authoring (a lazy policy now wins, or `intended` now
  loses). Catches benchmark rot.
- **Empirical** — from pilot/full-run data, flag packs where *every*
  model wins (too easy / a trivial idiom dominates — task #43) or
  *every* model loses (unsolvable or a predicate is mis-tuned —
  task #44). Discriminative packs are the only useful ones.

Paper payoff: a post-hoc audit table converts the "no-defect bar"
claim into something you can *show* — a strong methodology subsection.

### 12.2 Coverage map — RTS phase × decision-divergence
Map all 200 packs (plus the 1v1 battleground) onto the **RTS phase ×
decision-divergence matrix** from the original plan
(opening / early-mid / mid / mid-late / late × the canonical decisions
in each). Produce a coverage heatmap; flag empty / thin cells. Surface
the `meta.capability`-tag imbalance (`adversarial`=1 pack — full
end-to-end macro lives in the 1v1 battleground, both belong on the
map). Paper payoff: a figure showing the bench spans the *real* game,
not just easy probes.

### 12.3 Multi-run reliability — `pass^k`
Each (cell, seed) is run N times varying only model nondeterminism
(requires temperature > 0). Report mean ± CI **and** `pass^k`
(all-k-wins). A model that wins 5/10 identical runs is a fundamentally
different finding than 10/10. `--repeats N` in `run_eval`; default
`k=5` (Codex / SWE-bench convention). Paper payoff: mean-only is
fragile; reliability is itself a possible headline result.

## 13. Stretch ideas

- **Pivotal-turn analysis** — single-turn counterfactual swaps to show
  RTS losses are 1–2 catastrophic decisions, not uniform decay.

---

## 13. Ablation infrastructure already built (this is real, today)

- **Fog axis** — engine `reveal_map` no-fog flag (`OpenRA-Rust`),
  the `-clear` perception cells.
- **Modality axis** — `structured` / `vision` / `image` (image-
  primary, text redacted, labelled minimap) channels;
  `run_eval --perception-sweep` expands `pack:level` into the 6
  modality cells.
- **Handoff axis** — `openra_bench/handoff.py`
  (`HandoffController`, `TrajectoryController`), `run_eval
  --handoff-sweep`; the `passivity` metric on every result.
- **1v1 battleground + ELO** — `one_v_one.py`, scripted ladder.
- **Human-labeling** — the Play tab persists human runs in the
  standard `Playback` format (apples-to-apples with model runs).
- **200 scenario packs** — no-cheat-validated, capability-anchored.

---

## 14. Open decisions

- Model roster for the sweeps (which models; vision-capable required
  for `vision` / `image` channels).
- Compute / API budget for the full sweeps.
- Human-study scope (how many humans, which scenario subset).
- SFT base model(s) and the small/large pairing for finding 3.
- Strategy-classifier definition for §5.