# OpenRA-Bench — Scenario Design-Quality Audit & Scoring

Analysis only. No scenario files changed. Prepared 2026-05-17.
Scope: the 48 `status: active` packs (every pack without
`status: quarantine`); TEMPLATE and all `cat-c*-02..05` quarantine
packs excluded. Grounded against `SCENARIO_AUDIT.md`,
`win_conditions.py`, `scoring.py`, `goal_tracker.py`,
`eval_core.run_level`, the schema, and `openra_env.game_data`
(verified unit/building costs, prerequisites, tech tree).

---

## 1. Executive Summary

The capability taxonomy and predicate grammar are sound, and the
hand-authored families (perception-*, reasoning-*, action-*,
artofwar-*, strict-*) are mostly genuine, well-grounded decision
problems. But the active set has four structural quality problems that
materially weaken it as a discriminating benchmark:

1. **Loss is impossible in ~70% of active packs.** Only a
   `fail_condition` can produce `loss`; `run_level` leaves the outcome
   at `draw` when the win predicate is unmet (eval_core.py L174, L243).
   Every perception-*, reasoning-*, action-*, economy-*, custom-map,
   strict-sequence, building-and-planning, and most `cat-*` pack has
   **no real fail_condition** (the `cat-*` ones carry
   `units_lost_lte: -1`, which is *never* satisfiable because
   `units_lost = max(0, …) ≥ 0` — a dead no-op). Consequence: a
   do-nothing agent and a near-miss agent both score `draw` (0.5
   outcome); the binary outcome only separates win vs. not-win.
   Partial credit via `objective_progress` rescues *some*
   discrimination, but the headline outcome axis is degenerate on the
   majority of the suite.

2. **The `cat-c*` `-00`/`-01` kept representatives are near-exact
   duplicates, not distinct decisions.** `diff cat-c1-00 cat-c1-01`
   differs only in id/title and ±1 `explored_pct` / ±200 ticks.
   c5↔c6, c7↔c8, c9↔c11, c4↔c3 are the same decision with relabelled
   meta. The two kept variants per category add **zero** distinct
   capability — keep one, cut the second.

3. **`always-movement-wins` degeneracy in the navigation/perception
   `cat-*` and several hand-authored packs.** Where the only win
   clauses are `explored_pct_gte` + `within_ticks` with no fail and a
   generous clock, the *baseline scripted frontier agent*
   (`scripted_explore_agent`) already wins — these do not separate
   strong from weak models, only "moves" from "doesn't move."

4. **A genuine solvability defect in `strict-production-bom` hard**:
   the budget cannot cover the required tech path (see §2). Several
   "spend/build" packs are solvable but only test "spend the obvious
   thing," not an allocation trade-off, because the engine has no
   harvest income (documented S0/S1) so there is never scarcity
   pressure beyond a single fixed `starting_cash`.

**Bottom line:** ~9 packs should be CUT now (8 redundant `cat-*`
duplicates + 1 unsolvable level needs a budget fix or cut), ~16 need a
FIX (almost all: "add a real fail_condition so loss≠draw"), and the
hand-authored perception/reasoning/artofwar/strict-bom core is KEEP.
The single highest-leverage repo-wide fix is adding a real
`fail_condition` (timeout-loss or attrition-loss) to every active pack
so the outcome axis stops collapsing loss into draw.

---

## 2. Solvability spot-checks (verified against game_data)

Verified building/unit costs & prereqs: `powr` 300 (no prereq, +100
pwr), `tent`/`barr` 500 (prereq `powr`, −20 pwr), `proc` 1400 (prereq
`powr`), `pbox` 600 (prereq `tent`), `gun` 800 (prereq `weap`),
`tsla` 1200 (prereq `weap`, −75 pwr), `weap` War Factory (~2000),
`fact` 2000, e1 100 / e3 300 (prereq `barr|tent`).

- **strict-production-bom HARD — UNSOLVABLE on budget.** Spec:
  `tsla` + 3 e1 + 2 e3, `starting_cash: 4200`, agent faction soviet.
  `tsla` requires `weap` (≈2000) which is **not pre-placed and not in
  the actor list**. Minimum spend: barr 500 + weap 2000 + tsla 1200 +
  3·e1 300 + 2·e3 600 = **4600 > 4200**. Also tsla is −75 power with
  only one `powr` (+100) minus barr/weap drain → likely power-starved.
  → **FIX (raise hard `starting_cash` to ~5200 and pre-place or
  budget `weap`) or CUT the hard level.** Easy/medium are solvable
  (barr 500 + 3 e1; barr + 3 e1 + 2 e3 within 2200/3000) and a good
  BFCL-style fidelity test.
- **building-and-planning** easy (build powr+tent, cash 6000),
  medium (2 pbox in region, tent pre-placed, cash 5000: 2·600+powr),
  hard (MCV→deploy fact, 2 buildings near (60,20), cash 4000):
  all solvable; hard is tight but feasible. KEEP.
- **economy-investment / economy-time-box / economy-force-buildup**:
  all levels solvable on the given `starting_cash` (2nd proc 1400 +
  2nd powr 300 ≤ budgets; e1 at 100 makes `own_units_gte` trivially
  affordable). Solvable but **decision-thin** — no harvest income
  means "wide vs deep" is not a real trade (both fit the budget); the
  test reduces to "issue the build commands." FIX (see §4).
- **adversarial-* / artofwar-* / reasoning-risk-route /
  reasoning-frontier-commit / perception-***: policies exist
  (kite/focus-fire; bait then run; take the safe detour; commit to
  the correct fog region). Solvable; these are the strongest packs.
- **cat-c3/c4** (`building_total_gte` + `power_surplus_gte: 0`):
  solvable (powr is +100, free of prereq; spam powr). But
  `power_surplus_gte: 0` + `building_total_gte` is trivially gamed by
  building only power plants — see Rigor notes.
- **rush-hour / strategy-gauntlet / strategy-twobody / strategy-
  dilemma**: 14 scattered enemy infantry vs ~32 agent combat units,
  `units_killed_gte` 3–9; solvable, but with stance-3 auto-fighting
  units the agent barely has to act — see Discrimination.

---

## 3. Scored Table (1–5; 5 = best)

solv = solvability · disc = discrimination · grnd = grounding ·
rigor = non-gameable + principled scaling · ovr = overall

| pack id | capability | solv | disc | grnd | rigor | ovr | verdict | one-line note |
|---|---|---|---|---|---|---|---|---|
| action-multiunit-coordination | action | 5 | 4 | 5 | 4 | **4** | KEEP | Real parallel-control test; add fail_condition so non-win≠draw. |
| action-sequenced-execution | action | 5 | 4 | 5 | 4 | **4** | KEEP | after_ticks gates encode ordering well; add timeout fail_condition. |
| adversarial-duel | adversarial | 4 | 4 | 5 | 4 | **4** | KEEP | Has real fail (own_units_gte:1); reactive enemy = unique RTS value. |
| adversarial-siege | adversarial | 4 | 4 | 5 | 4 | **4** | KEEP | Same model as duel; entrenched defender variant — distinct enough. |
| adversarial-skirmish | adversarial | 4 | 4 | 5 | 4 | **4** | KEEP | Outnumbered/maneuver variant; genuine asymmetric tactic. |
| artofwar-decoy-sacrifice | reasoning | 4 | 5 | 5 | 5 | **5** | KEEP | Loss cap forces "spend bait, save army" — true credit assignment. |
| artofwar-indirect-approach | reasoning | 4 | 5 | 5 | 5 | **5** | KEEP | units_lost_lte:0 + hazard short path = clean long-horizon test. |
| artofwar-lure-the-tiger | reasoning | 3 | 4 | 5 | 4 | **4** | FIX | Two-phase intent good, but win has no clause proving the lure happened — reaching region by any survivable route also wins; add an ordering/after_ticks gate. |
| artofwar-sequenced-citadel | reasoning | 5 | 4 | 5 | 4 | **4** | KEEP | after_ticks+within_ticks band genuinely grades order; solid. |
| building-and-planning | reasoning | 4 | 4 | 5 | 4 | **4** | KEEP | 3 genuinely different decisions (tech dep / placement / relocate); add fail_condition. |
| custom-map-no-enemy | perception | 5 | 2 | 4 | 2 | **2** | FIX | Pure nav, no enemy, no fail → baseline move-agent wins easy/medium; only hard (all_units_in_region, tight clock) discriminates. Keep hard, cut/merge easy+medium or add fail. |
| economy-force-buildup | reasoning | 5 | 2 | 3 | 2 | **2** | FIX | e1@100 makes own_units_gte trivial; no scarcity (no harvest). Tests "click build," not allocation. Make win require a building mix + fail on idle. |
| economy-investment | reasoning | 5 | 3 | 4 | 3 | **3** | FIX | Best of the economy packs but "wide vs deep" isn't a real trade w/o income; both paths fit budget. Tighten cash so paths are mutually exclusive; add fail. |
| economy-time-box | reasoning | 5 | 2 | 3 | 2 | **2** | FIX | Near-duplicate of force-buildup + a building_total bar; same triviality. Merge into economy-investment or harden. |
| perception-frontier-reading | perception | 5 | 3 | 5 | 3 | **3** | FIX | Strong concept; but easy/medium = explored_pct only, no fail → baseline frontier agent already wins. Hard (decoys + units_lost_lte:1) is the real test. Add fail to easy/medium. |
| perception-target-vs-fog | perception | 5 | 4 | 5 | 4 | **4** | KEEP | buildings_discovered (not coverage) defeats the blind-sweep agent; medium/hard force inference. Add timeout fail. |
| reasoning-frontier-commit | reasoning | 5 | 4 | 5 | 4 | **4** | KEEP | Single scout = un-hedgeable commitment; decoy buildings vs units is a clean discriminator. |
| reasoning-risk-route | reasoning | 5 | 5 | 5 | 5 | **5** | KEEP | units_lost_lte:0 + lethal short path + clock = textbook non-gameable risk/route. Exemplary. |
| rush-hour | action | 5 | 2 | 3 | 2 | **2** | FIX | ~32 stance-3 auto-fighting agent units vs 14 scattered enemy; kill 3/6/9 in 40 turns is near-automatic. Low discrimination; not load-bearing. Cut to a real sweep (fewer units, coverage+kill, fail on timeout). |
| strategy-dilemma | reasoning | 4 | 2 | 3 | 2 | **2** | FIX | Win = buildings_discovered_gte:1 only → any scout that bumps the enemy base wins; "risk dilemma" framing not enforced by the predicate. Replace win with reach_region + units_lost_lte. |
| strategy-gauntlet | reasoning | 4 | 2 | 3 | 2 | **2** | FIX | Same defect: buildings_discovered_gte:1 with a huge friendly force; corridor never has to be solved. Re-spec to reach_region + attrition. |
| strategy-twobody | action | 4 | 2 | 3 | 2 | **2** | FIX | Two pre-split friendly blobs already near the enemy base; discover-1-building wins with no real coordination. Re-spec to all_units_in_region (true rendezvous). |
| strict-production-bom | action | 3 | 5 | 5 | 5 | **4** | FIX | Excellent BFCL-style exact-spec test (unit_type_count_eq + overproduction fail). HARD level unsolvable on 4200 cash (needs weap+tsla≈4600) — raise cash or cut hard. |
| strict-sequence | action | 5 | 4 | 5 | 4 | **4** | KEEP | Tool allowlist + after_ticks band genuinely tests API-fidelity ordering; add timeout fail so a stalled agent loses, not draws. |
| cat-c1-00 (Frontier Scouting) | perception | 5 | 3 | 4 | 3 | **3** | KEEP | Representative of C1; explored_pct+clock, hard adds units_lost_lte:0. OK as the single C1 kept level. |
| cat-c1-01 | perception | 5 | 2 | 4 | 2 | **2** | CUT | ±1 explored_pct / ±200 ticks vs cat-c1-00. Pure duplicate. |
| cat-c2-00 (Threat Enumeration) | perception | 5 | 3 | 4 | 3 | **3** | KEEP | enemies_discovered + units_lost_lte is a real perception-under-decoy test; keep one. |
| cat-c2-01 | perception | 5 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c2-00 (only tick deltas). |
| cat-c3-00 (Tech Critical Path) | reasoning | 4 | 2 | 4 | 2 | **2** | FIX | building_total_gte + power_surplus_gte:0 is gamed by spamming powr (each +100, no prereq) — doesn't test the tech *path*. Require has_building tent/proc to force the dependency. |
| cat-c3-01 | reasoning | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c3-00. |
| cat-c4-00 (Power-Budget Online) | reasoning | 4 | 2 | 4 | 2 | **2** | FIX | Same gameability as c3 (powr-spam satisfies both clauses); near-identical to c3. Merge c3+c4 into one fixed pack. |
| cat-c4-01 | reasoning | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c4-00 (and of c3). |
| cat-c5-00 (Budget Allocation) | reasoning | 5 | 2 | 3 | 2 | **2** | FIX | powr n:2 + building_total bar, no income → "build 5 cheap buildings"; not an indivisible-budget trade. Same as c6. |
| cat-c5-01 | reasoning | 5 | 2 | 3 | 2 | **2** | CUT | Duplicate of cat-c5-00. |
| cat-c6-00 (Time-Boxed Deploy) | reasoning | 5 | 2 | 3 | 2 | **2** | FIX | own_units_gte+building_total; e1@100 trivial; duplicate decision of c5/economy-time-box. Merge. |
| cat-c6-01 | reasoning | 5 | 2 | 3 | 2 | **2** | CUT | Duplicate of cat-c6-00. |
| cat-c7-00 (Defensive-Direction) | perception | 4 | 3 | 4 | 3 | **3** | KEEP | building_in_region for pbox forces a directional commit; pbox needs tent (pre-placed) — solvable. Keep one. |
| cat-c7-01 | perception | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c7-00. |
| cat-c8-00 (Base-Placement) | perception | 4 | 3 | 4 | 3 | **3** | FIX | Same family as c7 (place building in inferred region). Distinct enough to keep one but merge c7/c8 conceptually; ensure region is reachable/buildable. |
| cat-c8-01 | perception | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c8-00. |
| cat-c9-00 (Commit-vs-Retreat) | reasoning | 4 | 3 | 4 | 3 | **3** | KEEP | units_killed + units_lost_lte is a real risk call; no real fail (dead −1) — FIX-lite: make fail = units_lost_gte cap. |
| cat-c9-01 | reasoning | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c9-00. |
| cat-c10-00 (Force Coordination) | action | 5 | 3 | 4 | 3 | **3** | KEEP | all_units_in_region + units_lost_lte = genuine rendezvous; better-spec'd than strategy-twobody. Keep one. |
| cat-c10-01 | action | 5 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c10-00. |
| cat-c11-00 (Tempo Window) | reasoning | 4 | 3 | 4 | 3 | **3** | KEEP | after_ticks + units_killed + within_ticks genuinely encodes a window; only tempo coverage — keep one. |
| cat-c11-01 | reasoning | 4 | 2 | 4 | 2 | **2** | CUT | Duplicate of cat-c11-00 (and decision-twin of c9). |
| cat-c12-00 (Error Recovery) | reasoning | 4 | 2 | 3 | 2 | **2** | FIX | "Replan after setback" but nothing is destroyed at start — it's just a timed build with after_ticks:1200. Not a recovery test; either script a mid-episode loss or CUT. |
| cat-c12-01 | reasoning | 4 | 2 | 3 | 2 | **2** | CUT | Duplicate of cat-c12-00. |

Aggregate: KEEP 19 · FIX 18 · CUT 11. (TEMPLATE excluded.)

---

## 4. CUT list (concrete — safe to delete now)

These add no distinct capability and/or are non-discriminating:

1. **`cat-c1-01`** — byte-near duplicate of cat-c1-00 (±1 pct/±200 tk).
2. **`cat-c2-01`** — duplicate of cat-c2-00.
3. **`cat-c3-01`** — duplicate of cat-c3-00.
4. **`cat-c4-01`** — duplicate of cat-c4-00 (and decision-twin of c3).
5. **`cat-c5-01`** — duplicate of cat-c5-00.
6. **`cat-c6-01`** — duplicate of cat-c6-00 (decision-twin of c5).
7. **`cat-c7-01`** — duplicate of cat-c7-00.
8. **`cat-c8-01`** — duplicate of cat-c8-00.
9. **`cat-c9-01`** — duplicate of cat-c9-00.
10. **`cat-c10-01`** — duplicate of cat-c10-00.
11. **`cat-c11-01`** — duplicate of cat-c11-00.
12. **`cat-c12-01`** — duplicate of cat-c12-00.

Net: keep exactly one representative per `cat-c*` category (the `-00`),
deleting all 12 `-01` actives — removes 12 files with zero capability
loss. (Category-merge candidates, can be done after: fold c4→c3, c6→c5,
c8→c7, c11→c9 since each pair is the same predicate decision.)

Additionally **CUT or hard-FIX**: `cat-c12-00` (not actually an
error-recovery test — nothing fails mid-episode; it is a delayed timed
build), and `strict-production-bom` HARD level (unsolvable on 4200
cash; either raise to ~5200 + pre-place `weap`, or delete the hard
level and keep easy/medium).

---

## 5. The repo-wide FIX (do this first, highest leverage)

**Add a real `fail_condition` to every active pack that lacks one.**
Today `run_level` only emits `loss` if a `fail_condition` evaluates
true; otherwise an unmet win stays `draw` (eval_core L174/L243). With
no fail, a do-nothing agent draws and a 95%-there agent draws — the
outcome axis cannot rank effort. The `cat-*` `units_lost_lte: -1`
"fail" is a dead no-op (units_lost ≥ 0 always).

Mechanical, non-gameable fixes using the *existing* vocabulary:

- Timed tasks (perception-*, reasoning-*, action-*, economy-*,
  strict-*, building-and-planning, cat-c1/c3/c4/c5/c6/c7/c8/c12):
  `fail_condition: {not: {within_ticks: <max_ticks>}}` — i.e. running
  out the clock without meeting the win is a *loss*, not a draw.
- Attrition tasks (anything with a `units_lost_lte: N` win clause):
  add `fail_condition: {not: {units_lost_lte: N}}` so exceeding the
  cap loses immediately (also prunes wasted ticks).
- `cat-*`: replace the dead `units_lost_lte: -1` with the timeout-loss
  above (one-line generator change; regenerate the kept `-00` files).

This single change restores a 3-way outcome (win/draw/loss) across the
suite and makes `objective_progress` partial credit meaningful instead
of every non-win collapsing to 0.5.

Secondary FIXes (per pack, already noted in the table):

- **strategy-dilemma / strategy-gauntlet / strategy-twobody**: win is
  `buildings_discovered_gte: 1` — satisfied by *seeing* the pre-placed
  enemy base, which a huge friendly stance-3 force does incidentally.
  Re-spec to `reach_region` (the objective) + `units_lost_lte`
  (enforce the risk/coordination the title claims).
- **cat-c3/c4**: add `has_building: tent` (or `proc`) to the win so
  `building_total_gte` can't be met by powr-spam; this restores the
  tech-*dependency* the pack claims to test.
- **economy-investment / -time-box / -force-buildup**: tighten
  `starting_cash` so the wide and deep paths are mutually exclusive
  (currently both fit), and add the timeout fail. Until engine S0/S1
  (ore income) lands these can only ever be "spend the obvious thing."
- **artofwar-lure-the-tiger**: add an `after_ticks` gate or a
  staging `reach_region` on the lure so the win actually requires the
  two-phase plan, not just any survivable route to the objective.
- **rush-hour**: drop it from the "strategy" claim; reduce the
  friendly force and require coverage (`explored_pct_gte`) + kills +
  timeout-fail so it tests coordinated sweep, not auto-battle.

---

## 6. ADD list (top 10 new scenarios to author)

Gaps the active set still has, with win/fail sketches in the existing
predicate grammar and the external benchmark each parallels.

1. **adversarial-counterstrategy-read** (capability: adversarial).
   Opponent commits one of two observable openings (rush near-lane vs
   tech far-corner); agent must scout then commit the dominant
   counter. Win: `all_of[{buildings_discovered_gte:1},{units_killed_gte:N},{within_ticks:T}]`;
   fail: `{not:{own_units_gte:1}}` or timeout. Parallel:
   StarCraft-II-Arena, game-theoretic LLM evals. (Fills the
   opponent-modeling gap — the unique RTS value prop.)

2. **longhorizon-opening-to-assault** (reasoning). One episode,
   40k+ ticks: scout (explored_pct) → tech (`has_building: proc`) →
   army (`own_units_gte`) → strike (`reach_region` enemy base) — all
   four as `all_of`, terminal-only credit, timeout fail. Parallel:
   Terminal-Bench 2.0 / OSWorld 50-step. (Fills long-horizon credit
   assignment — currently thin.)

3. **strict-toolban-fidelity** (action). Achieve `reach_region` but
   the allowlist excludes `attack*`; win additionally requires
   `units_lost_lte: 0` AND a `not{after_ticks:T0}`-then-`reach`
   ordering. Any disallowed-tool call → fail. Parallel: BFCL V4
   relevance / τ²-bench. (Isolates API fidelity as the objective.)

4. **economy-throughput-real** (reasoning) — *gated on engine S0/S1*.
   Once ore income exists: `economy_value_gte: V within_ticks:T`,
   wide (2 proc) vs deep (1 proc + harvesters) genuinely mutually
   exclusive. Parallel: PlanBench cost-optimal. (Replaces the 5
   non-discriminating economy/cat-c5/c6 packs with one real test.)

5. **perception-count-the-threat** (perception). Win:
   `enemies_discovered_gte: K` where K = exact hidden count and
   over/under-scouting wastes the clock; fail timeout +
   `units_lost_lte`. Parallel: ERQA state-estimation. (Strengthens
   the validated transfer target with a counting variant.)

6. **reasoning-replan-after-loss** (reasoning) — a *true* error-
   recovery pack (what cat-c12 only pretends to be). Pre-place the
   agent base in the path of a scripted stance-3 enemy push that
   destroys 1–2 buildings early; win =
   `all_of[{building_total_gte:N},{has_building:tent},{after_ticks:t_loss},{within_ticks:T}]`.
   Parallel: PlanBench replanning. (Requires interrupt/scripted-loss
   support — see §7.)

7. **adversarial-feint-handling** (adversarial). Opponent shows a
   decoy force on one axis; win keyed to committing against the
   *real* axis (`building_in_region`/`units_killed_gte` at the true
   threat), losing if you commit to the decoy (timeout). Parallel:
   opponent modeling / adversarial robustness.

8. **coordination-staggered-window** (action). Two squads must each
   be in *different* regions within *different* tick bands
   (`all_of` of two `reach_region` + paired `after_ticks`/`within_ticks`),
   forcing true parallel scheduling, not a single column. Parallel:
   Watch-And-Help / multi-robot dispatch. (Upgrades the thin
   strategy-twobody/cat-c10 coordination bucket.)

9. **tempo-double-window** (reasoning). Two separate strike windows
   in one episode (`after_ticks:t1`+`units_killed_gte:a` then a
   forced lull then `after_ticks:t2`+`units_killed_gte:b`), so acting
   continuously fails. Parallel: TextStarCraft II tempo. (Tempo is
   currently a single cat-c11 category.)

10. **navigation-confined-hard-only** (perception). Replace the
    trivial custom-map easy/medium with one hard map-read:
    `all_units_in_region` in a far bounded corner, tight clock,
    `fail_condition` on timeout — a real ARC-/ERQA-style spatial
    commit with no degenerate easy tier.

---

## 7. Predicate-vocabulary additions needed

The current leaf set is adequate for §6 #1,2,5,7,8,9,10. To fully
author the others, add:

- **`tool_called`/`tool_not_called: <name>`** (or a scenario-level
  `forbidden_tools` enforced as an automatic `fail`) — needed for
  #3 strict-toolban-fidelity; today the only tool-fidelity signal is
  the cross-cutting `actions_warned/issued` score, not a win/fail
  predicate. This is the most leaderboard-relevant missing primitive
  (BFCL/τ²-bench relevance detection).
- **`building_destroyed_gte` / `own_building_lost_gte`** — needed for
  #6 (detect the scripted setback) and to give offensive packs a real
  "took the base" predicate instead of proxying with
  `buildings_discovered`/`reach_region`.
- **An ordering composite (e.g. `then: [A, B]` meaning A must have
  held true at some tick strictly before B)** — `after_ticks` only
  approximates sequencing by wall-clock; a true happened-before would
  let artofwar-lure-the-tiger / sequenced-citadel grade *order*
  rather than *timing*, and make #2/#6 non-gameable.
- A scripted-adversary / mid-episode-event hook (engine side, already
  flagged in SCENARIO_AUDIT as Phase-1) is the prerequisite for #1,
  #6, #7 to be more than scripted-stance encounters.

---

## 8. Prioritized action summary

**Cut now (12 files, zero capability loss):** all `cat-c*-01` actives
(c1–c12 `-01`). Optionally also retire `cat-c12-00` (mislabelled) and
the `strict-production-bom` hard level (unsolvable) unless budgeted.

**Fix now (one mechanical pass):** add a real `fail_condition`
(timeout-loss; attrition-loss where a `units_lost_lte` win clause
exists) to all ~36 active packs lacking one, and replace the dead
`units_lost_lte:-1` in the kept `cat-*` files. This is the single
change that most improves benchmark discrimination.

**Fix next (per-pack, ~6 packs):** strategy-dilemma/gauntlet/twobody
(win predicate doesn't enforce the claimed decision), cat-c3/c4
(powr-spam gameable — add `has_building`), economy-* (tighten cash so
allocation is a real trade), artofwar-lure-the-tiger (add ordering
gate), rush-hour (re-spec or stop counting it as strategy),
strict-production-bom hard (raise budget or cut).

**Top 10 to add (ranked):** #1 adversarial-counterstrategy-read,
#2 longhorizon-opening-to-assault, #3 strict-toolban-fidelity,
#6 reasoning-replan-after-loss, #7 adversarial-feint-handling,
#8 coordination-staggered-window, #5 perception-count-the-threat,
#9 tempo-double-window, #4 economy-throughput-real (after S0/S1),
#10 navigation-confined-hard-only.

**Predicate additions (in priority order):** (1) tool-call
fidelity predicate / `forbidden_tools` auto-fail; (2)
`building_destroyed_gte`; (3) a happened-before ordering composite;
(4) scripted mid-episode adversary hook (engine, Phase-1).

---
## De-duplication (overlap analysis, 24→21 active)

Win-predicate Jaccard overstates overlap (small vocab); judged at the
decision level. Consolidated the genuinely-aimless dups (same map +
same win signature + same skill):

- **adversarial-siege, adversarial-skirmish → quarantined**:
  same as adversarial-duel (rush-hour-arena, kill-reactive-enemy-within
  -ticks-with-loss-cap); duel's easy→hard ladder + easy/medium configs
  already cover even→outnumbered→entrenched. adversarial-duel is the
  canonical adversarial scenario.
- **economy-time-box → quarantined**: = economy-force-buildup + a
  "keep a building standing" clause (same budget-under-clock decision).
  Kept economy-force-buildup + economy-investment (the latter adds a
  distinct economy-vs-military allocation choice).

Kept (NOT aimless dup — each serves a purpose):
- strategy-dilemma vs strategy-gauntlet: same objective but **different
  maps** (singles-dilemma vs singles-gauntlet) → different route puzzle.
- artofwar-decoy-sacrifice vs -lure-the-tiger: same map but genuinely
  distinct credit-assignment (sacrifice-and-lose vs displace-and-exploit).

Active set: 21 distinct decisions. Quarantined packs remain on disk,
runnable via explicit --packs, excluded from the default sweep.

## Per-scenario closer look — #1 action-multiunit-coordination

Difficulty axis (one new controlled variable per tier, so a tier
failure attributes to a single capability):

- **easy** — 2 regions, exact `(x,y)`. Tests *parallel dispatch*:
  serialized one-group-at-a-time touring blows the binding tick
  deadline (~30+ turns); only genuine simultaneous control (~17–20
  turns) makes it. Split enforced by `units_in_region_gte n=2` (not
  `reach_region`, which one touring unit satisfies).
- **medium** — 3 *dispersed* regions (NE / SW bottom-left / SE),
  exact `(x,y)`, attrition cap, infantry pickets on the two eastern
  lanes. Adds *multi-vector* dispatch: a single eastward column
  cannot reach the SW region.
- **hard** — byte-identical setup to medium (same regions, pickets,
  deadline, attrition); the ONLY changed variable is
  `objective_coords: relative` — the briefing gives compass labels
  ("the NORTH-EAST corner") instead of coordinates. Tests *spatial
  grounding*: the model must localize each target on the minimap
  (each region holds one enemy building as the on-map landmark).
  Deliberately NOT spawn-varied and NO turrets on the path —
  introducing either would add a second uncontrolled variable and
  break the clean medium→hard attribution (recorded as a reasoned
  exception to the task-#23 spawn-variation contract in
  `tests/test_hard_tier.py::NOT_APPLICABLE`).

`objective_coords` (`exact|relative`, schema `Level`/`ScenarioConfig`,
default `exact`) is a reusable knob: any pack can opt into
coordinate-blind objectives. The win predicate always evaluates
against the real engine coordinates regardless of disclosure.

## Per-scenario closer look — #2 action-sequenced-execution

Original defect: the pack claimed to test ordered, non-stalling route
execution but the win condition referenced only the *final* region +
a tick band — a beeline-to-final that skipped every waypoint and
idled to the gate won (verified: model arrived tick 1353, idled ~900
ticks, won). Not model compensation: the scenario didn't enforce what
it advertised.

Fix — new reusable **stateful** predicate `waypoint_sequence`: latches
ordered region-visit progress (monotonic) on a per-episode
`signals.seq_progress` scratch. Wk+1 only counts after Wk; skip /
out-of-order / idle ⇒ never satisfied. Coords-aware phrase
(`exact|relative`). Difficulty axis (one new controlled variable per
tier):

- **easy** — ONE ordered 3-waypoint route, clear lanes, generous
  budget. Run: WIN, W1→W2→W3 in order, tick 1893 < 2400.
- **medium** — TWO ordered routes that *cross*, BOTH required ⇒ must
  split into two parallel columns (serial overruns); attrition cap.
  Run: LOSS (valid) — model split correctly, finished route S in
  order, lost column discipline on N and timed out (the exact
  re-deliberation failure the pack targets).
- **hard** — TWO 4-waypoint serpentine routes on the larger
  `scout-arena` (176×80, `scripts/build_scout_arena_map.py`), given
  ONLY by relative *search bands* (no coords); fogged markers revealed
  by the `enemy_building_spotted` interrupt; seed-varied 2 spawn
  points. Run: LOSS (valid) — machinery all verified working
  (interrupts fired on the big map); model drove to literal corners
  instead of scanning the bands, ignored the revealed positions,
  attrition-bust. Solvable in principle; not compensated.

All tick budgets aligned to ~90 ticks/turn so timeout / attrition /
wipe are reachable **losses** within `max_turns` (no draw degeneracy).
Footgun recorded: a `base_map` placed at the Level level is silently
ignored — it must go inside `overrides:` (guard test added).

## Win-speed bonus (cross-cutting scoring)

Every episode records `win_tick` / `win_turns` / `win_budget` /
`speed` / `composite_base` (score.json + manifest). On a **win** the
composite gets `+SPEED_BONUS·speed` (`SPEED_BONUS=0.05`,
`speed = 1 − win_tick/budget`, budget = tightest `within_ticks` in the
win tree else `max_ticks`). Bounded so a fast win ranks above a slow
win but never lifts a loss above a win or overrides correctness.
Leaderboard carries `win_speed`/`win_turns` and breaks composite/
win-rate ties by faster wins.

## YAML-referenceable generators (mapgen / botgen)

A pack no longer needs a static `.oramap` + static enemy actors:

- **Map** — `base_map: {generator: arena, name: scout-arena,
  width: 176, height: 80, cordon: 4}` materializes a deterministic,
  content-addressed `.oramap` at compile time (`openra_bench/mapgen.py`;
  static string ids still work). #2 hard uses this.
- **Bot** — `enemy: {faction: soviet, bot_type: hunt}` (alias `bot:`)
  attaches an engine-side scripted opponent. Behaviours (map-agnostic,
  ground-truth, per-tick): `hunt` (each unit attacks nearest agent
  unit), `rusher` (concentrated relentless charge), `patrol` (oscillate
  around own spawn, engage intruders), `turtle` (hold spawn, return
  fire). Runs in `openra-sim/scripted_bot` because the Python boundary
  can't see fogged enemy ids or command the enemy player; the bench
  surface (`openra_bench/botgen.py`) validates the name at compile.

## Whole-suite no-cheat pass (#1–#21, all 21 active packs)

Every active pack has been individually closer-looked + redesigned to
the **"no defect, no cheat"** bar: a scripted brute / stall / greedy /
shortest-path / sweep / wrong-route policy must LOSE on every level +
every hard seed; the intended capability policy must WIN; non-win is a
real reachable timeout LOSS, not a draw. Engine smoke clean across
seeds 1–4 each level (no panics, in-bounds, `map_supported`); hard
tiers in `tests/test_hard_tier.py::UPGRADED` keep ≥2 distinct seed-
driven spawns; reasoned `NOT_APPLICABLE` exceptions documented in the
test. Suite **414 passed, 1 skipped, 0 failed** on the consolidated
`main` after all 21 redesigns landed. Author-side artifacts: the
SCENARIO_REVIEW_CHECKLIST.md is the methodology; per-scenario commit
messages on `main` document each capability, the cheat path that was
closed, and the scripted before/after evidence.

Recurring defect classes the pass eliminated:
- **Inert deadlines** (`within_ticks` ≫ tick reachable at `max_turns`,
  i.e. > `93+90·(max_turns-1)`) ⇒ draw degeneracy. Every level now
  satisfies `within_ticks` AND `after_ticks` ≤ that bound.
- **Missing `fail_condition`** ⇒ non-win = draw. Every level now has
  an `any_of` fail covering at least the timeout (plus force-loss /
  attrition where applicable).
- **Capability not enforced by the win predicate** (the laziest play
  satisfies it without doing the advertised skill). Fixed with the
  right predicate per pack: `units_in_region_gte` (split / main body),
  `waypoint_sequence` (ordered / latched), `units_lost_lte` (bait
  budget), `buildings_discovered_gte` (fog-dependent discovery),
  `enemy_key_buildings_destroyed` (faithful "destroy fact+proc"),
  exact-cost `starting_cash` (binding allocation), etc.
- **Off-map actors** (engine panic). Every level's actors verified in
  playable bounds across seeds 1–4.
- **Engine auto-`done` on enemy-elimination** terminating before the
  intended win/fail evaluated → drew. Mitigation: an unarmed high-HP
  enemy `fact` marker at the objective ensures the episode runs to a
  real win or timeout LOSS.

Engine footguns surfaced during the pass (flagged for follow-up):
- **`power_surplus_gte` is currently inert**: the obs reports
  `power_provided/power_drained = 0` even with placed `powr`, so the
  predicate is effectively `0 ≥ N` and never fires meaningfully.
  Affected packs (economy-investment, tech-production-planning, etc.)
  intentionally do NOT rely on it as the sole discriminator.
- **`deploy` is not executed** by the installed wheel: an MCV reaches
  the target cell but never becomes a Construction Yard. `building-
  and-planning` hard was redesigned around build-radius creep instead.
- Both are tracked as engine-side follow-ups (separate from the
  pre-existing pending task #11 sabotage/specials).

Note: `capture_actor` is deliberately absent until the engine gains a
Capture order (pending S8 / task #11) — the bench never advertises a
tool the engine can't execute (1:1 parity).