Adversarial 1v1 spotlight: ladder family + rating + Elo wiring

- new 'adversarial' capability bucket (schema Literal + leaderboard
CAPABILITIES) — the head-to-head axis an RTS engine uniquely owns
- 3 ladder packs (adversarial-duel/-skirmish/-siege), each a 3-rung
easy→medium→hard escalation of a reactive engine-driven opponent;
per-level actor overrides give real rungs; win=destroy force within
ticks, fail=force wiped, hard adds a loss cap
- adversarial.py: pure ladder_rating (contiguous rungs cleared from
easy; all-seeds-won required) + adversarial_summary
- run_eval emits out['adversarial']; leaderboard ingest records
adversarial_rating + per-pack ladders; app capability table column;
Elo path stays via existing comparative pairwise on shared rungs
- documented swap-in to model-vs-model once the engine exposes an
enemy command channel (pairwise.py / task #3)
- tests: test_adversarial.py (6, incl live-engine pack runs) —
313 passed, 1 skipped

SCENARIO_AUDIT.md

@@ -0,0 +1,354 @@
+# OpenRA-Bench — Scenario Audit & Gap Analysis
+
+Research + audit. No code changes. Prepared 2026-05-17.
+
+---
+
+## Executive Summary
+
+OpenRA-Bench currently ships **85 scenario files** (17 hand-authored
+scenario families + 68 procedurally generated `cat-*` packs across 12
+categories, ~200 difficulty levels + 1 TEMPLATE). The design is
+unusually well-grounded: every win condition is a verifiable predicate,
+difficulty scales by *decision hardness* (info down, decoys up, clock
+down, attrition cap) rather than raw numbers, and each pack carries a
+`real_world_meaning` + `robotics_analogue`. The anti-memorization
+discipline (procedural variation, held-out seeds, generalization-gap
+metric) is consistent with current best practice (Procgen, SMACv2,
+ARC-AGI). The "RTS reasoning transfers" hypothesis is *directly
+supported by published evidence*: lmgame-Bench (arXiv:2505.15146)
+showed RL on Sokoban/Tetris lifted Blocksworld (+≥10pts planning) and
+WebShop (+6pts multi-turn decisions) but **not** math/coding — i.e.
+transfer is real but axis-specific (spatial / planning / embodied),
+exactly the axes this suite targets.
+
+**The core finding of this audit:** the capability *taxonomy* is
+sound and the transfer story is defensible, but the **coverage is
+badly skewed**. Frontier models in 2026 are differentiated almost
+entirely on (a) agentic **tool-use / function-calling fidelity** under
+strict APIs (BFCL V4, τ²-bench — GPT-5.5 reports 98% τ²-Telecom,
+Kimi-K2.6 50% Toolathlon), (b) **long-horizon multi-step agentic
+execution** (Terminal-Bench 2.0 — GPT-5.5 82.7%, SWE-bench Verified
+~89%), and (c) **fluid/abstract + spatial reasoning** (ARC-AGI-2 — the
+single largest 2026 model delta; ERQA for embodied/spatial). OpenRA-Bench
+is *strong* on spatial perception/exploration (FRONT/PERC — parallels
+ERQA, the validated transfer target) and *adequate* on constrained
+planning (PLAN/TECH/ECON — parallels PlanBench/Blocksworld). It is
+**weak on adversarial/game-theoretic reasoning** (no opponent-modeling
+scenario despite an RTS engine — the one capability an RTS bench is
+uniquely positioned to own), **weak on long-horizon credit assignment**
+(most clocks are short single-phase tasks, not 30+ step chains like
+Terminal-Bench/OSWorld), and **structurally weak on the very thing
+2026 leaderboards weight most: tool-use/instruction-following fidelity
+under a strict action API** — this is *measured* (action-validity
+sub-score) but no scenario *isolates* it as the primary objective.
+
+Additionally, ~68 of 85 packs are the auto-generated `cat-*` family:
+12 categories × ~3 levels × ~variants. Many `cat-c5/c6/c7/c8`
+"budget-allocation / base-placement" packs are near-duplicates of each
+other and of the hand-authored `economy-*` and `building-and-planning`
+scenarios — high level count, low *distinct-capability* count. Several
+economy packs degenerate to identical tests because harvest income is
+0 on the only loadable map (documented engine-prereq S0/S1) — these
+are honest but currently non-discriminating.
+
+**Top recommendations (detail in §4):**
+1. **Add an adversarial/opponent-modeling family** (the unique RTS
+   value prop; parallels game-theoretic + StarCraft-II-Arena evals).
+   Activate the Phase-1 1v1 path — this is the highest-leverage gap.
+2. **Add a strict-action-API instruction-following family** isolating
+   tool-call fidelity (parallels BFCL V4 / τ²-bench — the most
+   leaderboard-weighted 2026 capability).
+3. **Add 2–3 genuinely long-horizon scenarios** (multi-phase,
+   40k+ ticks, credit assignment across opening→tech→assault) to
+   parallel Terminal-Bench/OSWorld long-task tracks.
+4. **Cut/merge the redundant `cat-*` over-generation** to ~3 distinct
+   levels per *distinct* decision; the level *count* is inflating
+   apparent coverage without adding capability breadth.
+5. **Pre-register the transfer panel against ERQA (spatial/embodied),
+   Blocksworld/PlanBench (planning), and BFCL/τ²-bench (tool-use)** —
+   ERQA is the empirically-validated correlate for the headline claim;
+   BFCL validates the under-tested action-fidelity axis.
+
+---
+
+## STEP 1 — Scenario Inventory
+
+### 1.1 Counts
+
+| Group | Files | Notes |
+|---|---|---|
+| Hand-authored scenario families | 17 | perception/reasoning/action/strategy/economy |
+| Generated catalog packs (`cat-*`) | 68 | 12 categories (c1–c12), 5–6 variants × 3 levels each |
+| TEMPLATE | 1 | scaffold, not a scenario |
+| **Total scenario files** | **85** | ~200 difficulty levels |
+
+The 12 generated categories (from `SCENARIO_CATALOG.md`, verified
+against the `cat-*` files): C1 Frontier Scouting (FRONT), C2 Threat
+Enumeration (PERC), C3 Tech Critical Path (TECH), C4 Power-Budget
+Online (PLAN), C5 Budget Allocation (ECON), C6 Time-Boxed Capital
+Deploy (ECON), C7 Defensive-Direction Commit (PERC), C8 Base-Placement
+& Staging (PERC), C9 Commit-vs-Retreat (RISK), C10 Force Coordination
+(COORD), C11 Tempo/Timing Window (TEMPO), C12 Error Recovery/Replan
+(RISK).
+
+### 1.2 Capability buckets and what each actually tests
+
+The Perception→Reasoning→Action chain is the framing
+(`EVAL_STACK_PLAN.md`); every scenario is scored on all three links
+with a `weakest_link` diagnostic (`openra_bench/scoring.py`), plus a
+continuous `reward_vector` goal tracker (`openra_bench/goal_tracker.py`)
+and pass/fail via pure predicates (`openra_bench/scenarios/win_conditions.py`).
+
+| Bucket | Sub-skill | Scenarios | What it actually tests (predicate grammar) |
+|---|---|---|---|
+| **Perception** | spatial state-estimation under fog | `perception-frontier-reading`, `perception-target-vs-fog`, C2, C7, C8 | `explored_pct_gte`, `enemies/buildings_discovered_gte` under a clock + attrition cap. Read the minimap, steer sensing. |
+| **Perception/Front** | which-unexplored-region-to-commit | `reasoning-frontier-commit`, C1 | Frontier selection (pathfinding solved; *choosing* the region is the test). Strong ERQA/Active-Neural-SLAM analogue. |
+| **Reasoning** | constrained sequential planning | C3 (tech path), C4 (power budget), `building-and-planning` | Precondition-ordered build to a deadline with `power_surplus_gte≥0`. Blocksworld/PlanBench analogue. |
+| **Reasoning/Econ** | resource allocation / multi-objective | `economy-investment`, `economy-time-box`, `economy-force-buildup`, C5, C6, `economy-harvest-*` | Convert a fixed budget into units AND infrastructure under a clock; commit a wide-vs-deep allocation. Harvest variants currently non-discriminating (income=0, see §1.3). |
+| **Reasoning/Risk** | risk call / replan under partial info | `reasoning-risk-route`, `strategy-dilemma`, C9 (commit-vs-retreat), C12 (error recovery) | Safe-long vs short-lethal route; engage-vs-hold with attrition cap; rebuild after a setback (`after_ticks` gate). |
+| **Action** | multi-unit coordination / execution | `action-multiunit-coordination`, `strategy-twobody`, C10 | Drive dispersed squads to converge in a region with `units_lost_lte:0`. Watch-And-Help / SMAC analogue. |
+| **Action** | sequenced execution to deadline | `action-sequenced-execution`, `rush-hour`, `strategy-gauntlet` | Ordered build→deploy→reposition without dropping a step; multi-squad sweep-and-clear. |
+| **Tempo** | timing-window discipline | C11 | `after_ticks:t0` then `units_killed_gte:N` within `T1` — act in a window, not before. |
+
+All run on the single Rust-loadable map `rush-hour-arena`.
+
+### 1.3 Honest limitations already documented in-repo
+
+- **Harvest economy is non-functional** on the only loadable map: the
+  Python schema rejects `mine`/`gmine`, the `.oramap` files seed no
+  ore, and `silo` storage is hardcoded inert
+  (`SCENARIO_BRAINSTORM.md` §"Verified ground-truth constraints").
+  Consequence: `economy-harvest-timebox` / `economy-harvest-investment`
+  / harvest variants degenerate to spend-only tests identical to the
+  non-harvest economy packs. They are honestly tagged but **currently
+  non-discriminating** — they inflate the economy-bucket count without
+  adding a distinct capability. Blocked on engine-prereqs S0 (ore
+  source) + S1 (silo storage).
+- **No enemy-destruction predicate** — offensive success is proxied by
+  `units_killed_gte` + positional `reach_region`/`building_in_region`.
+- **No adversarial slot today** — the 1v1/Elo path is Phase-1, not
+  built (`EVAL_STACK_PLAN.md`). The "Elo leaderboard" is currently
+  only meaningful for the fixed-scenario composite, not head-to-head.
+
+---
+
+## STEP 2 — What Frontier Benchmarks Measure Today (2026)
+
+Verified via the four required sources + targeted search. URLs in §5.
+
+### 2.1 Required model pages
+
+- **Qwen3.6-35B-A3B** (HF page, released ~Apr 2026). Reports a
+  coding/agentic-heavy battery: SWE-bench Verified 73.4, SWE-bench
+  Pro 49.5, Terminal-Bench 2.0 51.5, **TAU3-Bench 67.2**, Tool
+  Decathlon 26.9, MCPMark 37.0; reasoning: MMLU-Pro 85.2, GPQA 86.0,
+  AIME26 92.7; **spatial-intelligence block: EmbSpatialBench 84.3,
+  RefSpatialBench 64.3, RefCOCO 92.0**. Takeaway: even a mid-size
+  model now reports a *dedicated embodied-spatial benchmark block* and
+  a *dedicated tool/agent block* — these are the differentiating axes.
+- **Kimi-K2.6** (HF page). Agentic-first: OSWorld-Verified 73.1,
+  **Toolathlon 50.0, MCPMark 55.9**, BrowseComp 83.2, Terminal-Bench
+  2.0 66.7, SWE-bench Verified 80.2; reasoning AIME 2026 96.4,
+  GPQA-Diamond 90.5. Tool-use / computer-use / long-horizon browsing
+  dominate the card.
+- **GPT-5.5** (OpenAI announcement; page is 403 to direct fetch,
+  corroborated via search). Released ~Apr 23 2026. Headline numbers:
+  **SWE-bench Verified 88.7**, SWE-bench Pro 58.6, **Terminal-Bench
+  2.0 82.7 (SOTA)**, **τ²-bench Telecom 98.0**, **ARC-AGI-2 85.0
+  (+11.7 over prior, the launch's single largest delta)**, GDPval
+  84.9. The two most-emphasized capabilities: agentic
+  coding/tool-orchestration and ARC-AGI-2 fluid reasoning.
+- **Artificial Analysis** leaderboard. The page no longer exposes
+  components inline, but the current **Intelligence Index v4.0** =
+  10 evals: **GDPval-AA, τ²-Bench Telecom, Terminal-Bench Hard,
+  SciCode, AA-LCR (long-context reasoning), AA-Omniscience, IFBench
+  (instruction-following), Humanity's Last Exam, GPQA Diamond,
+  CritPt**. Note the heavy weighting toward *agentic tool-use*
+  (τ²-bench), *long-horizon terminal agency* (Terminal-Bench Hard),
+  and *instruction-following fidelity* (IFBench). Pass@1, ±<1% CI.
+
+### 2.2 Broadly-used benchmarks, the capability each isolates, and the
+real-world proxy
+
+| Benchmark | Isolates | Real-world proxy | How models differentiate (2026) |
+|---|---|---|---|
+| **SWE-bench Verified / Pro** | repo-level multi-file code editing to a passing test | autonomous coding agents | Top cluster ~80–89% Verified (GPT-5.5 88.7); Pro still <60% — the hard frontier |
+| **Terminal-Bench 2.0** | long-horizon shell agency: plan→iterate→tool-coordinate, deterministic check | CLI/devops automation agents | Wide spread: GPT-5.5 82.7 vs Qwen3.6 51.5 — strong separator |
+| **τ-bench / τ²-bench (Telecom)** | multi-turn **tool use under a strict policy/API**, dialog state | customer-service / API agents | Near-saturating at the top (GPT-5.5 98.0) but mid-tier collapses — fidelity cliff |
+| **BFCL V4** | function-calling: parallel/multiple selection, **relevance (when NOT to call)**, multi-turn, agentic memory | tool/function-calling fidelity | AST-graded; single-turn solved, multi-turn/relevance still discriminating |
+| **GAIA** | compound multi-step tool-use + reasoning across web/files | general assistant agents | Multi-hop chains expose planning/credit-assignment failures |
+| **OSWorld** | full GUI computer control, 15- vs 50-step tasks | computer-use agents | Long-task tracks isolate horizon length specifically |
+| **ARC-AGI-2** | fluid abstract pattern induction, anti-memorization | genuine generalization | Largest 2026 deltas; GPT-5.5 85.0 — the prestige reasoning metric |
+| **GPQA / AIME / MMLU-Pro / HLE** | hard knowledge + math reasoning | expert QA | Largely saturating top-end except HLE — *not* a transfer target for game RL (lmgame-Bench) |
+| **ERQA** | **embodied spatial / trajectory / state-estimation / multi-view / task reasoning** (400 MCQ VQA) | robotics perception+planning | CoT moves it only +4–6.5pts → *reasoning-shaping* finetune is what moves it; the validated OpenRA-Bench transfer target |
+| **PlanBench / Blocksworld** | precondition-ordered sequential planning, cost-optimal, replanning | task-graph scheduling, robot task planning | Mystery-Blocksworld defeats memorization; *the* planning correlate |
+| **lmgame-Bench (Sokoban/Tetris RL → transfer)** | game-RL → spatial/planning transfer | the OpenRA-Bench hypothesis itself | RL→Blocksworld +≥10, →WebShop +6; NOT →GSM8K/coding |
+| **SMAC → SMACv2** | multi-agent micro; the anti-memorization cautionary tale | strategy-game RL | Open-loop policy beat SMAC ⇒ procedural variation mandatory (the suite follows this) |
+
+Net 2026 picture: differentiation has moved off knowledge/math QA
+(saturating) onto **agentic tool-use fidelity, long-horizon execution,
+computer/embodied control, and fluid reasoning (ARC-AGI-2)**.
+
+---
+
+## STEP 3 — Capability ↔ Coverage ↔ Parallel Benchmark Gap Table
+
+| Capability / real-world use case | OpenRA-Bench coverage | Parallel benchmark | Verdict |
+|---|---|---|---|
+| Spatial state-estimation / perception under partial obs | `perception-target-vs-fog`, `perception-frontier-reading`, C2, C7, C8 | **ERQA** (spatial/state-est), SpatialVLM, VSR | **Well-covered** — and this is the empirically-validated transfer target |
+| Frontier exploration: which unknown region to commit | `reasoning-frontier-commit`, C1 | Active Neural SLAM, frontier exploration, ERQA-trajectory | **Well-covered** |
+| Constrained sequential planning (precondition graph + deadline) | C3, C4, `building-and-planning` | **PlanBench / Blocksworld**, ALFWorld | **Adequate** (parallels exist; single-map limits variety) |
+| Resource allocation / multi-objective under a budget | `economy-investment`, `economy-time-box`, C5, C6 | PlanBench cost-optimal, SmartPlay | **Adequate** (spend-only) — harvest variants **non-discriminating** (engine S0/S1) |
+| Risk assessment / replanning under partial info | `reasoning-risk-route`, `strategy-dilemma`, C9, C12 | PlanBench replanning, StarCraft-II-Arena | **Adequate** |
+| Long-horizon multi-step credit assignment | thin: most clocks are short single-phase; C12 only gates with `after_ticks` | **Terminal-Bench 2.0**, OSWorld 50-step, GAIA | **Weak** — no genuine opening→tech→assault chain |
+| Multi-unit / multi-agent coordination | `action-multiunit-coordination`, `strategy-twobody`, C10 | Watch-And-Help, SMAC(v2) | **Adequate** |
+| Tool-use / function-calling **fidelity under a strict action API** | only as a *cross-cutting score* (`actions_warned/issued`); no scenario isolates it as the objective | **BFCL V4**, **τ²-bench**, IFBench | **Weak/structural gap** — yet this is the most leaderboard-weighted 2026 axis |
+| Instruction-following under strict constraints | implicit in win predicates; not isolated | **IFBench** (in AA Index v4) | **Weak** |
+| Adversarial / game-theoretic reasoning (opponent modeling, deception, counter-strategy) | **none** — no live adversary; `rush-hour` enemy is scripted; 1v1/Elo is Phase-1 unbuilt | StarCraft-II-Arena, game-theory evals; *the unique RTS value prop* | **Missing** — largest strategic gap |
+| Tempo / timing-window discipline | C11 | TextStarCraft II, SmartPlay | **Adequate** (but only 1 category, 15 levels) |
+| Fluid/abstract anti-memorization generalization | enforced by procedural variation + held-out seeds + generalization-gap metric | **ARC-AGI-2** philosophy | **Methodologically well-covered** (design discipline is correct) |
+
+### Scenarios with weak / no transfer story (flag)
+
+- **`rush-hour` / `strategy-gauntlet` "sweep-and-clear"** — primarily a
+  search-and-destroy execution task. Defensible as multi-robot
+  patrol-and-clear, but with a *scripted* enemy it tests execution,
+  not adversarial reasoning. Keep, but it should not be load-bearing
+  for the "strategy" claim.
+- **`economy-harvest-timebox` / `economy-harvest-investment`** —
+  currently economically identical to the non-harvest economy packs
+  (income=0). Honest, but they are *placeholders*, not distinct tests;
+  do not count them as economy coverage until S0/S1 land.
+- **`cat-c5/c6/c7/c8` over-generation** — many variants differ only in
+  region coordinates / target counts. The *decision* is the same; this
+  is level inflation, not capability breadth (mild
+  SMAC-style memorization risk if seeds aren't truly held out per
+  variant). Not "game trivia," but low marginal information.
+- The suite has **no** scenario that is pure game-trivia with zero
+  transfer story — the `real_world_meaning`/`robotics_analogue`
+  discipline is doing its job. The problem is *skew and redundancy*,
+  not arbitrariness.
+
+---
+
+## STEP 4 — Concrete, Prioritized Edit Advice
+
+### P0 — Add an adversarial / opponent-modeling family (highest leverage)
+
+An RTS engine's unique, defensible value vs every other benchmark is
+**live adversarial / game-theoretic reasoning** — and the suite
+currently has *zero*. Build the Phase-1 second RL-controlled slot
+(`EVAL_STACK_PLAN.md` Phase 1) and add:
+
+- **Counter-Strategy Read**: opponent commits an observable opening
+  (rush vs tech vs expand); agent must read it from partial recon and
+  pick the dominant counter. Capability: adversarial reasoning +
+  perception. Real-world: competitive multi-agent / negotiation /
+  red-teaming. Parallel: **StarCraft-II-Arena (ICLR'25)**,
+  game-theoretic LLM evals.
+- **Deception / feint handling**: opponent shows a decoy force; win
+  predicate rewards committing against the *real* axis
+  (`building_in_region`/`units_killed_gte` keyed off the true threat).
+  Parallel: adversarial robustness, opponent modeling.
+
+This also makes the **Elo leaderboard** genuinely meaningful
+(head-to-head), which it currently is not.
+
+### P0 — Add a strict-action-API instruction-following family
+
+The most leaderboard-weighted 2026 axis (τ²-bench in AA Index v4,
+BFCL V4, IFBench) is only measured as a *side diagnostic* here. Add a
+family whose **primary objective is action-API fidelity**:
+
+- Tasks where the win predicate is only reachable via a *specific
+  command sequence/format under explicit policy constraints* (e.g.
+  "achieve X but never issue `attack` before tick T", "use only
+  `move_units` + `deploy`, no `build`"), scoring relevance (issuing a
+  disallowed call = fail, à la BFCL "relevance detection").
+- Capability: instruction-following + tool-call fidelity under a
+  strict API. Real-world: agentic API/tool orchestration, policy
+  compliance. Parallel: **BFCL V4 (relevance/multi-turn), τ²-bench,
+  IFBench**.
+
+### P1 — Add 2–3 genuinely long-horizon, multi-phase scenarios
+
+Current clocks are mostly short single-phase. Add scenarios spanning
+opening→scout→economy→tech→army→engagement in **one episode (40k+
+ticks)** with a terminal objective, so the score depends on **credit
+assignment across phases** (an early scouting error must surface as a
+late failure). Capability: long-horizon credit assignment. Parallel:
+**Terminal-Bench 2.0, OSWorld 50-step track, GAIA** multi-hop. This is
+where 2026 models separate most and the suite is currently thin.
+
+### P1 — Cut / merge the redundant `cat-*` over-generation
+
+Reduce each generated category to **≤3 levels per genuinely distinct
+decision** instead of 5–6 near-duplicate coordinate variants. Net
+~200→~90 levels but *higher distinct-capability density* and lower
+memorization risk. Specifically merge: C5↔C6 (both "budget→units+
+buildings" ECON), C7↔C8 (both "place building in inferred region"
+PERC). Keep the procedural seed variation *within* a level (the
+anti-memorization mechanism) — cut the redundant *level* multiplication.
+
+### P1 — Quarantine non-discriminating economy packs
+
+Tag `economy-harvest-*` (and any harvest variant) as **engine-prereq /
+not-scored** until S0 (ore source) + S1 (silo storage) land
+(`SCENARIO_BRAINSTORM.md`). Do not count them in economy coverage
+claims. Land S0 (a one-line `VALID_ACTOR_TYPES` add or a
+`resource_fields` scenario field) — it is cheap and unblocks a real,
+distinct economy-throughput capability.
+
+### Over/under-representation summary
+
+- **Over-represented:** ECON spend-allocation (C5/C6 + 5 hand-authored
+  economy packs, several non-discriminating), and generated PERC
+  region-placement (C7/C8). Level count >> capability count.
+- **Under-represented:** adversarial/game-theoretic (zero),
+  tool-use/API-fidelity isolation (zero dedicated), long-horizon
+  credit assignment (thin), tempo (1 category).
+- **Right-sized:** spatial perception/frontier (the validated transfer
+  target — keep strong), coordination, risk/replan.
+
+### Strengthening the generalization-transfer argument
+
+The headline claim ("rush-hour finetune lifted ERQA") needs a
+pre-registered external transfer panel, scored as **per-axis deltas,
+not aggregate** (lmgame-Bench protocol):
+
+1. **ERQA** — the *primary* correlate. It is reasoning-sensitive
+   (CoT-only moves it just +4–6.5pts), embodied/spatial, and is the
+   axis the observed transfer hit. Report ERQA spatial / trajectory /
+   task subscores separately. Strongest evidence for the claim.
+2. **Blocksworld / PlanBench (incl. Mystery-Blocksworld)** —
+   validates the PLAN/TECH/ECON families; this is the exact transfer
+   lmgame-Bench demonstrated (RL→Blocksworld +≥10pts) and defeats
+   memorization.
+3. **BFCL V4 / τ²-bench** — validates the *new* P0 action-fidelity
+   family and tests whether strict-action-API discipline transfers
+   (the most commercially relevant axis; currently untested here).
+4. **Negative controls: GSM8K + a coding eval (e.g. LiveCodeBench)** —
+   lmgame-Bench showed game-RL does **not** transfer to math/coding;
+   include these to demonstrate the transfer is *specific* (spatial/
+   planning), not a generic capability bump. This negative result is
+   what makes the positive claim credible.
+5. Continue reporting the **generalization gap on held-out seeds**
+   (Procgen/SMACv2/ARC-AGI discipline) — already designed in; keep it
+   front-and-center as the anti-memorization guarantee.
+
+---
+
+## STEP 5 — Sources (verified via search/fetch, 2026-05)
+
+- Qwen3.6-35B-A3B model card — https://huggingface.co/Qwen/Qwen3.6-35B-A3B
+- Kimi-K2.6 model card — https://huggingface.co/moonshotai/Kimi-K2.6
+- Introducing GPT-5.5 — https://openai.com/index/introducing-gpt-5-5/ (direct fetch 403; corroborated via search: interestingengineering.com/ai-robotics/opanai-gpt-5-5-agentic-coding-gains, kingy.ai GPT-5.5 benchmarks, llm-stats.com/models/gpt-5.5)
+- Artificial Analysis leaderboard — https://artificialanalysis.ai/leaderboards/models ; methodology — https://artificialanalysis.ai/methodology/intelligence-benchmarking ; Intelligence Index — https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index
+- BFCL V4 — https://gorilla.cs.berkeley.edu/leaderboard.html ; paper https://openreview.net/forum?id=2GmDdhBdDk
+- τ²-bench (Sierra) via Agentic AI Benchmarks — https://awesomeagents.ai/leaderboards/agentic-ai-benchmarks-leaderboard/
+- ERQA — https://github.com/embodiedreasoning/ERQA ; Gemini Robotics arXiv:2503.20020 https://arxiv.org/html/2503.20020v1
+- lmgame-Bench arXiv:2505.15146 — https://arxiv.org/abs/2505.15146
+- GAIA / OSWorld / Terminal-Bench / ARC-AGI-2 overview — https://www.marktechpost.com/2026/04/26/top-7-benchmarks-that-actually-matter-for-agentic-reasoning-in-large-language-models/ ; https://www.spheron.network/blog/ai-agent-benchmarking-gpu-cloud-swebench-gaia/
+- In-repo: `SCENARIO_CATALOG.md`, `SCENARIO_BRAINSTORM.md`, `EVAL_STACK_PLAN.md`, `openra_bench/scenarios/win_conditions.py`, `openra_bench/scoring.py`, `openra_bench/goal_tracker.py`, `openra_bench/scenarios/packs/*.yaml`
+- Supporting literature cited in-repo and corroborated: SMAC→SMACv2 (arXiv:2212.07489), Procgen (arXiv:1912.01588), ARC-AGI (arXiv:1911.01547), PlanBench (arXiv:2206.10498), Active Neural SLAM (arXiv:2004.05155)

app.py

@@ -172,8 +172,9 @@ def load_capability_leaderboard() -> pd.DataFrame:
         rows = []
     cols = [
         "rank", "model", "episodes", "win_rate", "composite",
-        "objective", "perception", "reasoning", "action", "weakest_link",
-        "reward_vector", "held_out_composite", "generalization_gap",
+        "objective", "adversarial_rating", "perception", "reasoning",
+        "action", "weakest_link", "reward_vector",
+        "held_out_composite", "generalization_gap",
     ]
     if not rows:
         return pd.DataFrame(columns=cols)

openra_bench/adversarial.py

@@ -0,0 +1,72 @@
+"""Adversarial ladder rating (the 1v1 spotlight metric).
+
+Each `adversarial-*` pack is a 3-rung ladder (easy → medium → hard) of
+increasing reactive-opponent strength. A model's **ladder rating** on a
+pack is the number of rungs cleared *contiguously from the bottom* — a
+monotone difficulty signal that complements the Elo (which ranks models
+head-to-head on shared rungs via `pairwise.pairwise_elo`).
+
+Pure + deterministic; the live opponent is the engine's reactive force
+today, with a documented swap-in to model-vs-model once the engine
+exposes an enemy command channel (see pairwise.py / task #3).
+"""
+
+from __future__ import annotations
+
+RUNGS: tuple[str, ...] = ("easy", "medium", "hard")
+
+
+def ladder_rating(outcomes: dict[str, str]) -> int:
+    """Rungs cleared contiguously from easy. A rung is cleared iff its
+    outcome == "win". easy lost → 0; easy+medium won, hard lost → 2."""
+    n = 0
+    for r in RUNGS:
+        if outcomes.get(r) == "win":
+            n += 1
+        else:
+            break
+    return n
+
+
+def is_adversarial_episode(ep: dict) -> bool:
+    return ep.get("capability") == "adversarial"
+
+
+def ladder_ratings(stats: dict) -> dict[str, int]:
+    """Per adversarial pack → ladder rating, from a run_eval stats dict.
+
+    `cell` is "<pack>:<level>"; only the public split counts (held-out
+    seeds are anti-memorization, not ladder progression). When a rung
+    ran multiple seeds it is cleared only if it was won on *every*
+    seed (no lucky-seed promotion)."""
+    rungs: dict[str, dict[str, list[str]]] = {}
+    for e in stats.get("episodes", []):
+        if not is_adversarial_episode(e) or e.get("split", "public") != "public":
+            continue
+        pack, _, level = str(e.get("cell", "")).rpartition(":")
+        if not pack or level not in RUNGS:
+            continue
+        rungs.setdefault(pack, {}).setdefault(level, []).append(
+            e.get("outcome", "?")
+        )
+    out: dict[str, int] = {}
+    for pack, by_level in rungs.items():
+        collapsed = {
+            lv: ("win" if outs and all(o == "win" for o in outs) else "loss")
+            for lv, outs in by_level.items()
+        }
+        out[pack] = ladder_rating(collapsed)
+    return out
+
+
+def adversarial_summary(stats: dict) -> dict:
+    """Spotlight roll-up: per-pack ratings + the headline mean rating
+    (0–3) across adversarial packs played."""
+    ratings = ladder_ratings(stats)
+    mean = round(sum(ratings.values()) / len(ratings), 4) if ratings else 0.0
+    return {
+        "ladder_ratings": ratings,
+        "mean_ladder_rating": mean,
+        "packs": sorted(ratings),
+        "max_rung": len(RUNGS),
+    }

openra_bench/leaderboard.py

@@ -18,7 +18,7 @@ from collections import Counter
 from pathlib import Path
 from typing import Any
 
-CAPABILITIES = ("perception", "reasoning", "action")
+CAPABILITIES = ("perception", "reasoning", "action", "adversarial")
 DEFAULT_STORE = Path(__file__).parent.parent / "data" / "leaderboard.jsonl"
 # A run must cover at least this many episodes to be rankable (mirrors
 # the existing app.py min-games gate; keeps one-off noise off the board).
@@ -69,6 +69,13 @@ def ingest_run(
         # reward-vector signature, comparable across models/runs.
         "objective": overall.get("objective_mean", 0.0),
         "reward_vector": stats.get("reward_vector_mean", {}),
+        # Adversarial 1v1 spotlight: mean ladder rating (0–3) + per-pack.
+        "adversarial_rating": stats.get("adversarial", {}).get(
+            "mean_ladder_rating", 0.0
+        ),
+        "adversarial_ladders": stats.get("adversarial", {}).get(
+            "ladder_ratings", {}
+        ),
         "weakest_link_hist": overall.get("weakest_link_hist", {}),
         # Anti-memorization: held-out composite + generalization gap
         # (public − held-out). None when the run had no held-out split.

openra_bench/run_eval.py

@@ -202,6 +202,12 @@ def evaluate(
         "episodes": episodes,
         "skipped": skipped,
     }
+    # Adversarial spotlight: per-pack ladder ratings + headline mean.
+    from .adversarial import adversarial_summary
+
+    adv = adversarial_summary(out)
+    if adv["packs"]:
+        out["adversarial"] = adv
     if held_scores:
         ho = _agg(held_scores)
         out["overall_held_out"] = ho

openra_bench/scenarios/packs/adversarial-duel.yaml

@@ -0,0 +1,148 @@
+meta:
+  id: adversarial-duel
+  title: 'Adversarial: Force Duel'
+  capability: adversarial
+  real_world_meaning: "Defeat a reactive opposing force of escalating strength:\
+    \ focus-fire, target priority and kiting against an enemy that fights\
+    \ back — head-to-head tactical reasoning, not micro against a dummy."
+  robotics_analogue: "Adversarial multi-agent engagement: prevail over a\
+    \ reactive opponent team whose strength increases each ladder rung."
+  author: openra-bench
+base_map: rush-hour-arena
+# The ladder: easy/medium/hard are rungs of increasing enemy strength.
+# A model's adversarial rating = the highest rung it can clear (see
+# adversarial.ladder_rating); models are Elo-ranked on shared rungs.
+base:
+  agent:
+    faction: allies
+    cash: 0
+  enemy:
+    faction: soviet
+    cash: 0
+    bot_type: ''
+  actors:
+  - type: 2tnk
+    owner: agent
+    position: [6, 8]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 2
+    randomize:
+      position:
+        base: [6, 8]
+        offset: 2
+  - type: 1tnk
+    owner: agent
+    position: [6, 11]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 2
+  - type: e1
+    owner: enemy
+    position: [34, 22]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 3
+levels:
+  easy:
+    description: 'Rung 1 — even-ish duel: 4 agent vehicles vs 3 enemy rifle.'
+    win_condition:
+      all_of:
+      - units_killed_gte: 3
+      - within_ticks: 14000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 60
+  medium:
+    description: 'Rung 2 — outnumbered: same force vs a reactive armour mix.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 8]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+        randomize:
+          position:
+            base: [6, 8]
+            offset: 2
+      - type: 1tnk
+        owner: agent
+        position: [6, 11]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+      - type: e1
+        owner: enemy
+        position: [34, 22]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 4
+      - type: 1tnk
+        owner: enemy
+        position: [38, 26]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+    win_condition:
+      all_of:
+      - units_killed_gte: 6
+      - within_ticks: 12000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 70
+  hard:
+    description: 'Rung 3 — badly outnumbered + a loss cap: win by maneuver.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 8]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+        randomize:
+          position:
+            base: [6, 8]
+            offset: 2
+      - type: 1tnk
+        owner: agent
+        position: [6, 11]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+      - type: e1
+        owner: enemy
+        position: [34, 22]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 5
+      - type: 2tnk
+        owner: enemy
+        position: [40, 28]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 3
+    win_condition:
+      all_of:
+      - units_killed_gte: 8
+      - within_ticks: 10000
+      - units_lost_lte: 3
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 80

openra_bench/scenarios/packs/adversarial-siege.yaml

@@ -0,0 +1,112 @@
+meta:
+  id: adversarial-siege
+  title: 'Adversarial: Break the Position'
+  capability: adversarial
+  real_world_meaning: "Dislodge a reactive force holding a prepared\
+    \ position: commit, absorb the defended ground, and break it before\
+    \ the clock — assaulting an opponent with the terrain advantage."
+  robotics_analogue: "Adversarial objective seizure: overcome a reactive\
+    \ defending team entrenched at the goal under a time budget."
+  author: openra-bench
+base_map: rush-hour-arena
+base:
+  agent:
+    faction: allies
+    cash: 0
+  enemy:
+    faction: soviet
+    cash: 0
+    bot_type: ''
+  actors:
+  - type: 2tnk
+    owner: agent
+    position: [6, 10]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 4
+  - type: e1
+    owner: enemy
+    position: [40, 26]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 4
+levels:
+  easy:
+    description: 'Rung 1 — 4 tanks break a 4-rifle position.'
+    win_condition:
+      all_of:
+      - units_killed_gte: 4
+      - within_ticks: 13000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 60
+  medium:
+    description: 'Rung 2 — defenders reinforced with armour.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 10]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 4
+      - type: e1
+        owner: enemy
+        position: [40, 26]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 5
+      - type: 2tnk
+        owner: enemy
+        position: [43, 29]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+    win_condition:
+      all_of:
+      - units_killed_gte: 7
+      - within_ticks: 11000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 75
+  hard:
+    description: 'Rung 3 — dug-in mixed force, tight clock + loss cap.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 10]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 5
+      - type: e1
+        owner: enemy
+        position: [40, 26]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 6
+      - type: 2tnk
+        owner: enemy
+        position: [43, 29]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 4
+    win_condition:
+      all_of:
+      - units_killed_gte: 10
+      - within_ticks: 9000
+      - units_lost_lte: 3
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 85

openra_bench/scenarios/packs/adversarial-skirmish.yaml

@@ -0,0 +1,112 @@
+meta:
+  id: adversarial-skirmish
+  title: 'Adversarial: Outnumbered Skirmish'
+  capability: adversarial
+  real_world_meaning: "Win while outnumbered by trading space for time:\
+    \ pick fights, retreat, and re-engage a reactive enemy on favourable\
+    \ terms instead of a head-on brawl — asymmetric tactical reasoning."
+  robotics_analogue: "Asymmetric adversarial control: a smaller agent team\
+    \ must defeat a larger reactive force by engagement selection."
+  author: openra-bench
+base_map: rush-hour-arena
+base:
+  agent:
+    faction: allies
+    cash: 0
+  enemy:
+    faction: soviet
+    cash: 0
+    bot_type: ''
+  actors:
+  - type: 2tnk
+    owner: agent
+    position: [6, 9]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 2
+  - type: e1
+    owner: enemy
+    position: [30, 20]
+    stance: 3
+    health: 100
+    facing: -1
+    count: 4
+levels:
+  easy:
+    description: 'Rung 1 — 2 tanks vs 4 rifle: doable if you fight smart.'
+    win_condition:
+      all_of:
+      - units_killed_gte: 4
+      - within_ticks: 14000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 60
+  medium:
+    description: 'Rung 2 — 2 tanks vs 6 mixed: must split the enemy.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 9]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+      - type: e1
+        owner: enemy
+        position: [30, 20]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 4
+      - type: 1tnk
+        owner: enemy
+        position: [35, 24]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 2
+    win_condition:
+      all_of:
+      - units_killed_gte: 6
+      - within_ticks: 12000
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 70
+  hard:
+    description: 'Rung 3 — 3 tanks vs 8, loss-capped: maneuver or lose.'
+    overrides:
+      actors:
+      - type: 2tnk
+        owner: agent
+        position: [6, 9]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 3
+      - type: e1
+        owner: enemy
+        position: [30, 20]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 5
+      - type: 1tnk
+        owner: enemy
+        position: [35, 24]
+        stance: 3
+        health: 100
+        facing: -1
+        count: 3
+    win_condition:
+      all_of:
+      - units_killed_gte: 8
+      - within_ticks: 10000
+      - units_lost_lte: 2
+    fail_condition:
+      not:
+        own_units_gte: 1
+    max_turns: 80

openra_bench/scenarios/schema.py

@@ -26,7 +26,9 @@ from pydantic import BaseModel, Field, field_validator
 from .win_conditions import WinCondition
 
 LevelName = Literal["easy", "medium", "hard"]
-Capability = Literal["perception", "reasoning", "action"]
+# "adversarial" = head-to-head reasoning vs a reactive opponent (the
+# axis an RTS engine uniquely owns); ranked by a difficulty ladder + Elo.
+Capability = Literal["perception", "reasoning", "action", "adversarial"]
 
 
 def deep_merge(base: dict, patch: dict) -> dict:

tests/test_adversarial.py

@@ -0,0 +1,101 @@
+"""Adversarial 1v1 spotlight: ladder rating + family integ.
+
+The ladder metric is pure logic (fast, exhaustive); the 3 packs are
+also compiled and smoke-run on the live Rust engine to prove the new
+`adversarial` capability flows end to end (schema → engine → score →
+leaderboard breakdown).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from openra_bench.adversarial import (
+    RUNGS,
+    adversarial_summary,
+    ladder_rating,
+    ladder_ratings,
+)
+from openra_bench.leaderboard import _capability_breakdown, ingest_run
+
+PACKS = Path(__file__).parent.parent / "openra_bench" / "scenarios" / "packs"
+ADV = ["adversarial-duel", "adversarial-skirmish", "adversarial-siege"]
+
+
+def test_ladder_rating_is_contiguous_from_easy():
+    assert ladder_rating({}) == 0
+    assert ladder_rating({"easy": "loss"}) == 0
+    assert ladder_rating({"easy": "win"}) == 1
+    assert ladder_rating({"easy": "win", "medium": "win"}) == 2
+    assert ladder_rating(dict.fromkeys(RUNGS, "win")) == 3
+    # non-contiguous: hard won but medium lost → still 1
+    assert ladder_rating(
+        {"easy": "win", "medium": "loss", "hard": "win"}
+    ) == 1
+    # draw does not clear a rung
+    assert ladder_rating({"easy": "draw"}) == 0
+
+
+def test_ladder_ratings_need_all_seeds_won():
+    stats = {
+        "episodes": [
+            {"cell": "adversarial-duel:easy", "capability": "adversarial",
+             "split": "public", "outcome": "win"},
+            {"cell": "adversarial-duel:easy", "capability": "adversarial",
+             "split": "public", "outcome": "loss"},  # one seed lost
+            {"cell": "adversarial-duel:medium", "capability": "adversarial",
+             "split": "public", "outcome": "win"},
+            # non-adversarial + held-out ignored
+            {"cell": "rush-hour:easy", "capability": "action",
+             "split": "public", "outcome": "win"},
+            {"cell": "adversarial-duel:hard", "capability": "adversarial",
+             "split": "held_out", "outcome": "win"},
+        ]
+    }
+    # easy not all-won → rating 0 (medium can't count, non-contiguous)
+    assert ladder_ratings(stats) == {"adversarial-duel": 0}
+    s = adversarial_summary(stats)
+    assert s["packs"] == ["adversarial-duel"]
+    assert s["mean_ladder_rating"] == 0.0 and s["max_rung"] == 3
+
+
+def test_summary_and_leaderboard_carry_adversarial(tmp_path):
+    stats = {
+        "episodes": [
+            {"cell": "adversarial-duel:easy", "capability": "adversarial",
+             "split": "public", "outcome": "win", "composite": 0.7},
+            {"cell": "adversarial-duel:medium", "capability": "adversarial",
+             "split": "public", "outcome": "win", "composite": 0.6},
+            {"cell": "adversarial-duel:hard", "capability": "adversarial",
+             "split": "public", "outcome": "loss", "composite": 0.2},
+        ],
+        "overall": {"n": 3, "win_rate": 0.66, "composite_mean": 0.5},
+        "adversarial": None,
+    }
+    stats["adversarial"] = adversarial_summary(stats)
+    assert stats["adversarial"]["ladder_ratings"] == {"adversarial-duel": 2}
+    rec = ingest_run(stats, "m1", store=tmp_path / "lb.jsonl")
+    assert rec["adversarial_rating"] == 2 / 1  # mean over 1 pack
+    assert rec["adversarial_ladders"] == {"adversarial-duel": 2}
+    cap = _capability_breakdown(stats["episodes"])
+    assert "adversarial" in cap and cap["adversarial"]["n"] == 3
+
+
+@pytest.mark.parametrize("pid", ADV)
+def test_adversarial_pack_compiles_and_runs(pid):
+    pytest.importorskip("openra_train")
+    from openra_bench.eval_core import run_level
+    from openra_bench.scenarios import load_pack
+    from openra_bench.scenarios.loader import compile_level
+
+    pack = load_pack(PACKS / f"{pid}.yaml")
+    for lvl in RUNGS:
+        c = compile_level(pack, lvl)
+        assert c.meta.capability == "adversarial"
+        assert c.map_supported, f"{pid}:{lvl} map must be Rust-loadable"
+    c = compile_level(pack, "easy")
+    res = run_level(c, lambda rs, C: [C.observe()], seed=1)
+    assert res.outcome in {"win", "draw", "loss"}
+    assert res.turns >= 1