SCENARIO_CATALOG.md

> **RETIRED (curation decision).** The auto-generated 200-level `cat-*`
> corpus described below has been removed from the repo. It scored low
> on discrimination/rigor (see `SCENARIO_QUALITY.md`) and added
> memorization surface without distinct capability. The benchmark is now
> a small **curated, hand-authored** set (Art-of-War, economy/building
> planning, strategy, rush-hour, adversarial, strict-spec, perception,
> reasoning), each designed deliberately one at a time with a
> non-gameable win+fail and a named external-benchmark analogue. This
> file is kept only as historical design notes.

# OpenRA-Bench Scenario Catalog (research-grounded, 200 levels)

Goal: a benchmark whose scenarios **generalize on reasoning** — we
empirically observed finetuning on the rush-hour scenario also lifted
**ERQA**. The literature explains why and how to make that systematic.

## Why this transfers (literature)

- **Game/spatial RL → planning transfer is real but axis-specific.**
  lmgame-Bench (Hu et al., 2025, arXiv:2505.15146): RL on simplified
  Sokoban lifted Blocksworld-1D 17.9→32.7%, Blocksworld-2D 9.0→29.5%,
  WebShop 7.0→19.1% — but *not* GSM8K/SQL. Transfer targets are
  spatial / planning / embodied, **not** math.
- **RL not SFT.** "Understanding Transferability of LLM Reasoning"
  (arXiv:2507.00432): RL generalizes (minimal representation drift);
  trajectory SFT catastrophically forgets. Use these scenarios with
  verifiable-reward RL, not trajectory SFT.
- **ERQA is reasoning-sensitive.** ERQA (Gemini Robotics, DeepMind,
  arXiv:2503.20020): 400 MCQ over spatial / trajectory / action /
  state-estimation / multi-view / task reasoning; CoT alone moves it
  +4–6.5pts → a reasoning-shaping finetune (rush-hour) plausibly moves
  its spatial/trajectory/task subscores. Our suite is built to drive
  exactly those axes.
- **Anti-memorization is mandatory.** SMAC→SMACv2 (arXiv:2212.07489):
  a timestep/ID open-loop policy beat SMAC → memorization, not
  reasoning. Procgen (arXiv:1912.01588), Crafter (arXiv:2109.06780),
  ARC-AGI (arXiv:1911.01547): procedurally varied, held-out instances;
  report the **generalization gap**. Distractor/decoy decorrelation of
  "obvious cue" vs "correct action" (shortcut-learning, arXiv:2412.05152).

## Capability taxonomy → measurement

Consolidated from the general-reasoning, embodied/spatial, and
games-as-eval surveys. Metrics borrow canonical instruments: **SPL**
(Anderson et al., 2018, arXiv:1807.06757) for "reach/find under fog";
**coverage-under-budget / time-to-find / region-commitment regret**
(Active Neural SLAM, Chaplot et al., ICLR 2020, arXiv:2004.05155;
frontier exploration, Yamauchi 1997); **Goal-Condition Success +
path-length weighting** (ALFRED, arXiv:1912.01734); **predicate
satisfaction** (PlanBench, arXiv:2206.10498; BEHAVIOR-1K,
arXiv:2403.09227); **win/attrition under a difficulty ladder**
(TextStarCraft II, arXiv:2312.11865; SmartPlay, arXiv:2310.01557).

| Cap | Dimension | Primary literature |
|---|---|---|
| PERC | spatial/state-estimation under partial obs | ERQA; SpatialVLM (2401.12168); VSR (TACL'23) |
| FRONT | which-unexplored-region-to-commit | Active Neural SLAM; frontier exploration; exploration-aware EQA (2503.11117) |
| PLAN | sequential planning under constraints | PlanBench; ALFWorld (2010.03768) |
| TECH | precondition / tech-graph ordering | PlanBench (Mystery-Blocksworld); BEHAVIOR-1K BDDL |
| ECON | resource allocation / multi-objective | PlanBench cost-optimal; SmartPlay |
| COORD | multi-unit / multi-agent coordination | Watch-And-Help (2010.09890); SMAC |
| RISK | risk assessment / replanning under partial info | PlanBench replanning; StarCraft II Arena (ICLR'25) |
| TEMPO | timing / tempo windows | TextStarCraft II; SmartPlay |

## The 200-level catalog (12 categories, ~67 three-level packs)

From the engine-grounded end-to-end RA dissection (28 decision points,
phases opening→scouting→economy→tech→army→engagement→defense→assault→
endgame). Every win-condition uses ONLY predicates in
`win_conditions.py`; all on `rush-hour-arena` (only Rust-loadable map
until S10). Difficulty ladder = the *decision* gets harder (less info,
more decoys, tighter clock, committed-not-loose budget, attrition
cap) — never just bigger numbers (Procgen/SMACv2 rule).

| # | Category | Cap | Levels | Win pattern (real grammar) | Status |
|---|---|---|---|---|---|
| C1 | Frontier Scouting | FRONT | 18 | `all_of[explored_pct_gte:P,(units_lost_lte:0),within_ticks:T]` | now |
| C2 | Threat Enumeration | PERC | 18 | `all_of[enemies_discovered_gte:N / buildings_discovered_gte:K, units_lost_lte:0, within_ticks:T]` | now |
| C3 | Tech Critical Path | TECH | 18 | `all_of[has_building:X, building_total_gte:N, power_surplus_gte:0, within_ticks:T]` | now |
| C4 | Power-Budget Online | PLAN | 15 | `all_of[power_surplus_gte:0, building_total_gte:N, within_ticks:T]` | now |
| C5 | Budget Allocation (spend) | ECON | 18 | `all_of[building_count_gte{A,n}, building_count_gte{B,n}, within_ticks:T]` | now |
| C6 | Time-Boxed Capital Deploy | ECON | 18 | `all_of[own_units_gte:K, building_total_gte:M,(units_lost_lte:0),within_ticks:T]` | now |
| C7 | Defensive-Direction Commit | PERC | 18 | `all_of[building_in_region{type:pbox,...,count:k}, within_ticks:T]` | now |
| C8 | Base-Placement & Staging | PERC | 15 | `all_of[building_in_region{type:fact,...},(reach_region),within_ticks:T]` | now |
| C9 | Commit-vs-Retreat | RISK | 18 | `all_of[units_killed_gte:N, units_lost_lte:L, within_ticks:T]` | now |
| C10 | Force Coordination | COORD | 18 | `all_of[all_units_in_region{...}, units_lost_lte:0, within_ticks:T]` | now |
| C11 | Tempo / Timing Window | TEMPO | 15 | `all_of[after_ticks:t0, units_killed_gte:N, within_ticks:T1]` | now |
| C12 | Error Recovery / Replan | RISK | 15 | `all_of[has_building:X, building_total_gte:N, after_ticks:t0, within_ticks:T]` | now |

≈ **201 levels / ~67 packs**, all runnable on the current engine.
Economy categories are scoped as **spend/conservation** decisions
(`starting_cash`), the discriminating choice today; an
**Economy-Harvest** 13th category is **now RUNNABLE** — engine task
**#14** (S0 ore-seeding + idempotent harvest, S1 capped resource pool
from refineries/silos + `economy_value`) is done. Shipped packs:
`economy-harvest-timebox` (earn max economic value in a time budget,
keep harvesters productive) and `economy-harvest-investment` (split a
budget between widening collection and silo storage so burst yield is
not lost — the storage decision). Win = `economy_value_gte`
(=cash+stored resources) / `harvesters_gte` / `has_building:silo`
within `within_ticks`; integ-tested end-to-end (test_economy_harvest).

## Generation & integ-test strategy ("each run + validated")

- Packs are **generated** by `tools/gen_catalog.py` (procedurally
  parameterized per the anti-memorization rule), written flat into
  `openra_bench/scenarios/packs/` with a `cat-` prefix.
- **Validation**: `python -m openra_bench.scenarios.validate` compiles
  all 3 levels of every pack against the engine `ScenarioDefinition`
  + win-grammar (schema gate).
- **Engine run**: the existing parametrized
  `tests/test_robustness.py::test_bundled_pack_runs_on_engine`
  auto-covers every `packs/*.yaml` — each catalog pack's easy level
  must actually execute on the Rust engine (terrain resolves, actors
  load, scores), not merely validate.
- **Catalog coverage test**: `tests/test_catalog.py` asserts all 12
  categories are present, ≈200 levels exist, every pack carries a
  capability tag, and difficulty is monotone (tighter clock / less
  info per level) — the literature's controlled-ladder requirement.
- **Transfer panel (follow-up)**: pre-register ERQA + Blocksworld +
  WebShop external eval to quantify generalization gap (lmgame-Bench
  protocol); report per-axis deltas, not aggregate.

## Cross-cutting rules (enforced by the generator)

1. No enemy-destruction predicate exists → offensive success =
   `units_killed_gte` + positional `reach_region`/`building_in_region`.
2. Difficulty = decision hardness (info↓, decoys↑, clock↓, budget
   committed, attrition cap), never raw scaling.
3. Per-pack `real_world_meaning` + `robotics_analogue` are mandatory
   and tie to the mapped capability dimension + its literature.