SCENARIO_QUALITY.md

OpenRA-Bench — Scenario Design-Quality Audit & Scoring

Analysis only. No scenario files changed. Prepared 2026-05-17. Scope: the 48 status: active packs (every pack without status: quarantine); TEMPLATE and all cat-c*-02..05 quarantine packs excluded. Grounded against SCENARIO_AUDIT.md, win_conditions.py, scoring.py, goal_tracker.py, eval_core.run_level, the schema, and openra_env.game_data (verified unit/building costs, prerequisites, tech tree).

---

1. Executive Summary

The capability taxonomy and predicate grammar are sound, and the hand-authored families (perception-, reasoning-, action-, artofwar-, strict-*) are mostly genuine, well-grounded decision problems. But the active set has four structural quality problems that materially weaken it as a discriminating benchmark:

1. Loss is impossible in ~70% of active packs. Only a fail_condition can produce loss; run_level leaves the outcome at draw when the win predicate is unmet (eval_core.py L174, L243). Every perception-, reasoning-, action-, economy-, custom-map, strict-sequence, building-and-planning, and most cat-* pack has no real fail_condition (the cat-* ones carry units_lost_lte: -1, which is never satisfiable because units_lost = max(0, …) ≥ 0 — a dead no-op). Consequence: a do-nothing agent and a near-miss agent both score draw (0.5 outcome); the binary outcome only separates win vs. not-win. Partial credit via objective_progress rescues some discrimination, but the headline outcome axis is degenerate on the majority of the suite.

2. The cat-c* -00/-01 kept representatives are near-exact duplicates, not distinct decisions. diff cat-c1-00 cat-c1-01 differs only in id/title and ±1 explored_pct / ±200 ticks. c5↔c6, c7↔c8, c9↔c11, c4↔c3 are the same decision with relabelled meta. The two kept variants per category add zero distinct capability — keep one, cut the second.

3. always-movement-wins degeneracy in the navigation/perception cat-* and several hand-authored packs. Where the only win clauses are explored_pct_gte + within_ticks with no fail and a generous clock, the baseline scripted frontier agent (scripted_explore_agent) already wins — these do not separate strong from weak models, only "moves" from "doesn't move."

4. A genuine solvability defect in strict-production-bom hard: the budget cannot cover the required tech path (see §2). Several "spend/build" packs are solvable but only test "spend the obvious thing," not an allocation trade-off, because the engine has no harvest income (documented S0/S1) so there is never scarcity pressure beyond a single fixed starting_cash.

Bottom line: ~9 packs should be CUT now (8 redundant cat-* duplicates + 1 unsolvable level needs a budget fix or cut), ~16 need a FIX (almost all: "add a real fail_condition so loss≠draw"), and the hand-authored perception/reasoning/artofwar/strict-bom core is KEEP. The single highest-leverage repo-wide fix is adding a real fail_condition (timeout-loss or attrition-loss) to every active pack so the outcome axis stops collapsing loss into draw.

---

2. Solvability spot-checks (verified against game_data)

Verified building/unit costs & prereqs: powr 300 (no prereq, +100 pwr), tent/barr 500 (prereq powr, −20 pwr), proc 1400 (prereq powr), pbox 600 (prereq tent), gun 800 (prereq weap), tsla 1200 (prereq weap, −75 pwr), weap War Factory (~2000), fact 2000, e1 100 / e3 300 (prereq barr|tent).

• strict-production-bom HARD — UNSOLVABLE on budget. Spec: tsla + 3 e1 + 2 e3, starting_cash: 4200, agent faction soviet. tsla requires weap (≈2000) which is not pre-placed and not in the actor list. Minimum spend: barr 500 + weap 2000 + tsla 1200 + 3·e1 300 + 2·e3 600 = 4600 > 4200. Also tsla is −75 power with only one powr (+100) minus barr/weap drain → likely power-starved. → FIX (raise hard starting_cash to ~5200 and pre-place or budget weap) or CUT the hard level. Easy/medium are solvable (barr 500 + 3 e1; barr + 3 e1 + 2 e3 within 2200/3000) and a good BFCL-style fidelity test.
• building-and-planning easy (build powr+tent, cash 6000), medium (2 pbox in region, tent pre-placed, cash 5000: 2·600+powr), hard (MCV→deploy fact, 2 buildings near (60,20), cash 4000): all solvable; hard is tight but feasible. KEEP.
• economy-investment / economy-time-box / economy-force-buildup: all levels solvable on the given starting_cash (2nd proc 1400 + 2nd powr 300 ≤ budgets; e1 at 100 makes own_units_gte trivially affordable). Solvable but decision-thin — no harvest income means "wide vs deep" is not a real trade (both fit the budget); the test reduces to "issue the build commands." FIX (see §4).
• adversarial- / artofwar- / reasoning-risk-route / reasoning-frontier-commit / perception-***: policies exist (kite/focus-fire; bait then run; take the safe detour; commit to the correct fog region). Solvable; these are the strongest packs.
• cat-c3/c4 (building_total_gte + power_surplus_gte: 0): solvable (powr is +100, free of prereq; spam powr). But power_surplus_gte: 0 + building_total_gte is trivially gamed by building only power plants — see Rigor notes.
• rush-hour / strategy-gauntlet / strategy-twobody / strategy- dilemma: 14 scattered enemy infantry vs ~32 agent combat units, units_killed_gte 3–9; solvable, but with stance-3 auto-fighting units the agent barely has to act — see Discrimination.

---

3. Scored Table (1–5; 5 = best)

solv = solvability · disc = discrimination · grnd = grounding · rigor = non-gameable + principled scaling · ovr = overall

pack id	capability	solv	disc	grnd	rigor	ovr	verdict	one-line note
action-multiunit-coordination	action	5	4	5	4	4	KEEP	Real parallel-control test; add fail_condition so non-win≠draw.
action-sequenced-execution	action	5	4	5	4	4	KEEP	after_ticks gates encode ordering well; add timeout fail_condition.
adversarial-duel	adversarial	4	4	5	4	4	KEEP	Has real fail (own_units_gte:1); reactive enemy = unique RTS value.
adversarial-siege	adversarial	4	4	5	4	4	KEEP	Same model as duel; entrenched defender variant — distinct enough.
adversarial-skirmish	adversarial	4	4	5	4	4	KEEP	Outnumbered/maneuver variant; genuine asymmetric tactic.
artofwar-decoy-sacrifice	reasoning	4	5	5	5	5	KEEP	Loss cap forces "spend bait, save army" — true credit assignment.
artofwar-indirect-approach	reasoning	4	5	5	5	5	KEEP	units_lost_lte:0 + hazard short path = clean long-horizon test.
artofwar-lure-the-tiger	reasoning	3	4	5	4	4	FIX	Two-phase intent good, but win has no clause proving the lure happened — reaching region by any survivable route also wins; add an ordering/after_ticks gate.
artofwar-sequenced-citadel	reasoning	5	4	5	4	4	KEEP	after_ticks+within_ticks band genuinely grades order; solid.
building-and-planning	reasoning	4	4	5	4	4	KEEP	3 genuinely different decisions (tech dep / placement / relocate); add fail_condition.
custom-map-no-enemy	perception	5	2	4	2	2	FIX	Pure nav, no enemy, no fail → baseline move-agent wins easy/medium; only hard (all_units_in_region, tight clock) discriminates. Keep hard, cut/merge easy+medium or add fail.
economy-force-buildup	reasoning	5	2	3	2	2	FIX	e1@100 makes own_units_gte trivial; no scarcity (no harvest). Tests "click build," not allocation. Make win require a building mix + fail on idle.
economy-investment	reasoning	5	3	4	3	3	FIX	Best of the economy packs but "wide vs deep" isn't a real trade w/o income; both paths fit budget. Tighten cash so paths are mutually exclusive; add fail.
economy-time-box	reasoning	5	2	3	2	2	FIX	Near-duplicate of force-buildup + a building_total bar; same triviality. Merge into economy-investment or harden.
perception-frontier-reading	perception	5	3	5	3	3	FIX	Strong concept; but easy/medium = explored_pct only, no fail → baseline frontier agent already wins. Hard (decoys + units_lost_lte:1) is the real test. Add fail to easy/medium.
perception-target-vs-fog	perception	5	4	5	4	4	KEEP	buildings_discovered (not coverage) defeats the blind-sweep agent; medium/hard force inference. Add timeout fail.
reasoning-frontier-commit	reasoning	5	4	5	4	4	KEEP	Single scout = un-hedgeable commitment; decoy buildings vs units is a clean discriminator.
reasoning-risk-route	reasoning	5	5	5	5	5	KEEP	units_lost_lte:0 + lethal short path + clock = textbook non-gameable risk/route. Exemplary.
rush-hour	action	5	2	3	2	2	FIX	~32 stance-3 auto-fighting agent units vs 14 scattered enemy; kill 3/6/9 in 40 turns is near-automatic. Low discrimination; not load-bearing. Cut to a real sweep (fewer units, coverage+kill, fail on timeout).
strategy-dilemma	reasoning	4	2	3	2	2	FIX	Win = buildings_discovered_gte:1 only → any scout that bumps the enemy base wins; "risk dilemma" framing not enforced by the predicate. Replace win with reach_region + units_lost_lte.
strategy-gauntlet	reasoning	4	2	3	2	2	FIX	Same defect: buildings_discovered_gte:1 with a huge friendly force; corridor never has to be solved. Re-spec to reach_region + attrition.
strategy-twobody	action	4	2	3	2	2	FIX	Two pre-split friendly blobs already near the enemy base; discover-1-building wins with no real coordination. Re-spec to all_units_in_region (true rendezvous).
strict-production-bom	action	3	5	5	5	4	FIX	Excellent BFCL-style exact-spec test (unit_type_count_eq + overproduction fail). HARD level unsolvable on 4200 cash (needs weap+tsla≈4600) — raise cash or cut hard.
strict-sequence	action	5	4	5	4	4	KEEP	Tool allowlist + after_ticks band genuinely tests API-fidelity ordering; add timeout fail so a stalled agent loses, not draws.
cat-c1-00 (Frontier Scouting)	perception	5	3	4	3	3	KEEP	Representative of C1; explored_pct+clock, hard adds units_lost_lte:0. OK as the single C1 kept level.
cat-c1-01	perception	5	2	4	2	2	CUT	±1 explored_pct / ±200 ticks vs cat-c1-00. Pure duplicate.
cat-c2-00 (Threat Enumeration)	perception	5	3	4	3	3	KEEP	enemies_discovered + units_lost_lte is a real perception-under-decoy test; keep one.
cat-c2-01	perception	5	2	4	2	2	CUT	Duplicate of cat-c2-00 (only tick deltas).
cat-c3-00 (Tech Critical Path)	reasoning	4	2	4	2	2	FIX	building_total_gte + power_surplus_gte:0 is gamed by spamming powr (each +100, no prereq) — doesn't test the tech path. Require has_building tent/proc to force the dependency.
cat-c3-01	reasoning	4	2	4	2	2	CUT	Duplicate of cat-c3-00.
cat-c4-00 (Power-Budget Online)	reasoning	4	2	4	2	2	FIX	Same gameability as c3 (powr-spam satisfies both clauses); near-identical to c3. Merge c3+c4 into one fixed pack.
cat-c4-01	reasoning	4	2	4	2	2	CUT	Duplicate of cat-c4-00 (and of c3).
cat-c5-00 (Budget Allocation)	reasoning	5	2	3	2	2	FIX	powr n:2 + building_total bar, no income → "build 5 cheap buildings"; not an indivisible-budget trade. Same as c6.
cat-c5-01	reasoning	5	2	3	2	2	CUT	Duplicate of cat-c5-00.
cat-c6-00 (Time-Boxed Deploy)	reasoning	5	2	3	2	2	FIX	own_units_gte+building_total; e1@100 trivial; duplicate decision of c5/economy-time-box. Merge.
cat-c6-01	reasoning	5	2	3	2	2	CUT	Duplicate of cat-c6-00.
cat-c7-00 (Defensive-Direction)	perception	4	3	4	3	3	KEEP	building_in_region for pbox forces a directional commit; pbox needs tent (pre-placed) — solvable. Keep one.
cat-c7-01	perception	4	2	4	2	2	CUT	Duplicate of cat-c7-00.
cat-c8-00 (Base-Placement)	perception	4	3	4	3	3	FIX	Same family as c7 (place building in inferred region). Distinct enough to keep one but merge c7/c8 conceptually; ensure region is reachable/buildable.
cat-c8-01	perception	4	2	4	2	2	CUT	Duplicate of cat-c8-00.
cat-c9-00 (Commit-vs-Retreat)	reasoning	4	3	4	3	3	KEEP	units_killed + units_lost_lte is a real risk call; no real fail (dead −1) — FIX-lite: make fail = units_lost_gte cap.
cat-c9-01	reasoning	4	2	4	2	2	CUT	Duplicate of cat-c9-00.
cat-c10-00 (Force Coordination)	action	5	3	4	3	3	KEEP	all_units_in_region + units_lost_lte = genuine rendezvous; better-spec'd than strategy-twobody. Keep one.
cat-c10-01	action	5	2	4	2	2	CUT	Duplicate of cat-c10-00.
cat-c11-00 (Tempo Window)	reasoning	4	3	4	3	3	KEEP	after_ticks + units_killed + within_ticks genuinely encodes a window; only tempo coverage — keep one.
cat-c11-01	reasoning	4	2	4	2	2	CUT	Duplicate of cat-c11-00 (and decision-twin of c9).
cat-c12-00 (Error Recovery)	reasoning	4	2	3	2	2	FIX	"Replan after setback" but nothing is destroyed at start — it's just a timed build with after_ticks:1200. Not a recovery test; either script a mid-episode loss or CUT.
cat-c12-01	reasoning	4	2	3	2	2	CUT	Duplicate of cat-c12-00.

Aggregate: KEEP 19 · FIX 18 · CUT 11. (TEMPLATE excluded.)

---

4. CUT list (concrete — safe to delete now)

These add no distinct capability and/or are non-discriminating:

1. cat-c1-01 — byte-near duplicate of cat-c1-00 (±1 pct/±200 tk).
2. cat-c2-01 — duplicate of cat-c2-00.
3. cat-c3-01 — duplicate of cat-c3-00.
4. cat-c4-01 — duplicate of cat-c4-00 (and decision-twin of c3).
5. cat-c5-01 — duplicate of cat-c5-00.
6. cat-c6-01 — duplicate of cat-c6-00 (decision-twin of c5).
7. cat-c7-01 — duplicate of cat-c7-00.
8. cat-c8-01 — duplicate of cat-c8-00.
9. cat-c9-01 — duplicate of cat-c9-00.
10. cat-c10-01 — duplicate of cat-c10-00.
11. cat-c11-01 — duplicate of cat-c11-00.
12. cat-c12-01 — duplicate of cat-c12-00.

Net: keep exactly one representative per cat-c* category (the -00), deleting all 12 -01 actives — removes 12 files with zero capability loss. (Category-merge candidates, can be done after: fold c4→c3, c6→c5, c8→c7, c11→c9 since each pair is the same predicate decision.)

Additionally CUT or hard-FIX: cat-c12-00 (not actually an error-recovery test — nothing fails mid-episode; it is a delayed timed build), and strict-production-bom HARD level (unsolvable on 4200 cash; either raise to ~5200 + pre-place weap, or delete the hard level and keep easy/medium).

---

5. The repo-wide FIX (do this first, highest leverage)

Add a real fail_condition to every active pack that lacks one. Today run_level only emits loss if a fail_condition evaluates true; otherwise an unmet win stays draw (eval_core L174/L243). With no fail, a do-nothing agent draws and a 95%-there agent draws — the outcome axis cannot rank effort. The cat-* units_lost_lte: -1 "fail" is a dead no-op (units_lost ≥ 0 always).

Mechanical, non-gameable fixes using the existing vocabulary:

• Timed tasks (perception-, reasoning-, action-, economy-, strict-*, building-and-planning, cat-c1/c3/c4/c5/c6/c7/c8/c12): fail_condition: {not: {within_ticks: <max_ticks>}} — i.e. running out the clock without meeting the win is a loss, not a draw.
• Attrition tasks (anything with a units_lost_lte: N win clause): add fail_condition: {not: {units_lost_lte: N}} so exceeding the cap loses immediately (also prunes wasted ticks).
• cat-*: replace the dead units_lost_lte: -1 with the timeout-loss above (one-line generator change; regenerate the kept -00 files).

This single change restores a 3-way outcome (win/draw/loss) across the suite and makes objective_progress partial credit meaningful instead of every non-win collapsing to 0.5.

Secondary FIXes (per pack, already noted in the table):

• strategy-dilemma / strategy-gauntlet / strategy-twobody: win is buildings_discovered_gte: 1 — satisfied by seeing the pre-placed enemy base, which a huge friendly stance-3 force does incidentally. Re-spec to reach_region (the objective) + units_lost_lte (enforce the risk/coordination the title claims).
• cat-c3/c4: add has_building: tent (or proc) to the win so building_total_gte can't be met by powr-spam; this restores the tech-dependency the pack claims to test.
• economy-investment / -time-box / -force-buildup: tighten starting_cash so the wide and deep paths are mutually exclusive (currently both fit), and add the timeout fail. Until engine S0/S1 (ore income) lands these can only ever be "spend the obvious thing."
• artofwar-lure-the-tiger: add an after_ticks gate or a staging reach_region on the lure so the win actually requires the two-phase plan, not just any survivable route to the objective.
• rush-hour: drop it from the "strategy" claim; reduce the friendly force and require coverage (explored_pct_gte) + kills + timeout-fail so it tests coordinated sweep, not auto-battle.

---

6. ADD list (top 10 new scenarios to author)

Gaps the active set still has, with win/fail sketches in the existing predicate grammar and the external benchmark each parallels.

1. adversarial-counterstrategy-read (capability: adversarial). Opponent commits one of two observable openings (rush near-lane vs tech far-corner); agent must scout then commit the dominant counter. Win: all_of[{buildings_discovered_gte:1},{units_killed_gte:N},{within_ticks:T}]; fail: {not:{own_units_gte:1}} or timeout. Parallel: StarCraft-II-Arena, game-theoretic LLM evals. (Fills the opponent-modeling gap — the unique RTS value prop.)

2. longhorizon-opening-to-assault (reasoning). One episode, 40k+ ticks: scout (explored_pct) → tech (has_building: proc) → army (own_units_gte) → strike (reach_region enemy base) — all four as all_of, terminal-only credit, timeout fail. Parallel: Terminal-Bench 2.0 / OSWorld 50-step. (Fills long-horizon credit assignment — currently thin.)

3. strict-toolban-fidelity (action). Achieve reach_region but the allowlist excludes attack*; win additionally requires units_lost_lte: 0 AND a not{after_ticks:T0}-then-reach ordering. Any disallowed-tool call → fail. Parallel: BFCL V4 relevance / τ²-bench. (Isolates API fidelity as the objective.)

4. economy-throughput-real (reasoning) — gated on engine S0/S1. Once ore income exists: economy_value_gte: V within_ticks:T, wide (2 proc) vs deep (1 proc + harvesters) genuinely mutually exclusive. Parallel: PlanBench cost-optimal. (Replaces the 5 non-discriminating economy/cat-c5/c6 packs with one real test.)

5. perception-count-the-threat (perception). Win: enemies_discovered_gte: K where K = exact hidden count and over/under-scouting wastes the clock; fail timeout + units_lost_lte. Parallel: ERQA state-estimation. (Strengthens the validated transfer target with a counting variant.)

6. reasoning-replan-after-loss (reasoning) — a true error- recovery pack (what cat-c12 only pretends to be). Pre-place the agent base in the path of a scripted stance-3 enemy push that destroys 1–2 buildings early; win = all_of[{building_total_gte:N},{has_building:tent},{after_ticks:t_loss},{within_ticks:T}]. Parallel: PlanBench replanning. (Requires interrupt/scripted-loss support — see §7.)

7. adversarial-feint-handling (adversarial). Opponent shows a decoy force on one axis; win keyed to committing against the real axis (building_in_region/units_killed_gte at the true threat), losing if you commit to the decoy (timeout). Parallel: opponent modeling / adversarial robustness.

8. coordination-staggered-window (action). Two squads must each be in different regions within different tick bands (all_of of two reach_region + paired after_ticks/within_ticks), forcing true parallel scheduling, not a single column. Parallel: Watch-And-Help / multi-robot dispatch. (Upgrades the thin strategy-twobody/cat-c10 coordination bucket.)

9. tempo-double-window (reasoning). Two separate strike windows in one episode (after_ticks:t1+units_killed_gte:a then a forced lull then after_ticks:t2+units_killed_gte:b), so acting continuously fails. Parallel: TextStarCraft II tempo. (Tempo is currently a single cat-c11 category.)

10. navigation-confined-hard-only (perception). Replace the trivial custom-map easy/medium with one hard map-read: all_units_in_region in a far bounded corner, tight clock, fail_condition on timeout — a real ARC-/ERQA-style spatial commit with no degenerate easy tier.

---

7. Predicate-vocabulary additions needed

The current leaf set is adequate for §6 #1,2,5,7,8,9,10. To fully author the others, add:

• tool_called/tool_not_called: <name> (or a scenario-level forbidden_tools enforced as an automatic fail) — needed for #3 strict-toolban-fidelity; today the only tool-fidelity signal is the cross-cutting actions_warned/issued score, not a win/fail predicate. This is the most leaderboard-relevant missing primitive (BFCL/τ²-bench relevance detection).
• building_destroyed_gte / own_building_lost_gte — needed for #6 (detect the scripted setback) and to give offensive packs a real "took the base" predicate instead of proxying with buildings_discovered/reach_region.
• An ordering composite (e.g. then: [A, B] meaning A must have held true at some tick strictly before B) — after_ticks only approximates sequencing by wall-clock; a true happened-before would let artofwar-lure-the-tiger / sequenced-citadel grade order rather than timing, and make #2/#6 non-gameable.
• A scripted-adversary / mid-episode-event hook (engine side, already flagged in SCENARIO_AUDIT as Phase-1) is the prerequisite for #1, #6, #7 to be more than scripted-stance encounters.

---

8. Prioritized action summary

Cut now (12 files, zero capability loss): all cat-c*-01 actives (c1–c12 -01). Optionally also retire cat-c12-00 (mislabelled) and the strict-production-bom hard level (unsolvable) unless budgeted.

Fix now (one mechanical pass): add a real fail_condition (timeout-loss; attrition-loss where a units_lost_lte win clause exists) to all ~36 active packs lacking one, and replace the dead units_lost_lte:-1 in the kept cat-* files. This is the single change that most improves benchmark discrimination.

Fix next (per-pack, ~6 packs): strategy-dilemma/gauntlet/twobody (win predicate doesn't enforce the claimed decision), cat-c3/c4 (powr-spam gameable — add has_building), economy-* (tighten cash so allocation is a real trade), artofwar-lure-the-tiger (add ordering gate), rush-hour (re-spec or stop counting it as strategy), strict-production-bom hard (raise budget or cut).

Top 10 to add (ranked): #1 adversarial-counterstrategy-read, #2 longhorizon-opening-to-assault, #3 strict-toolban-fidelity, #6 reasoning-replan-after-loss, #7 adversarial-feint-handling, #8 coordination-staggered-window, #5 perception-count-the-threat, #9 tempo-double-window, #4 economy-throughput-real (after S0/S1), #10 navigation-confined-hard-only.

Predicate additions (in priority order): (1) tool-call fidelity predicate / forbidden_tools auto-fail; (2) building_destroyed_gte; (3) a happened-before ordering composite; (4) scripted mid-episode adversary hook (engine, Phase-1).

---

De-duplication (overlap analysis, 24→21 active)

Win-predicate Jaccard overstates overlap (small vocab); judged at the decision level. Consolidated the genuinely-aimless dups (same map + same win signature + same skill):

• adversarial-siege, adversarial-skirmish → quarantined: same as adversarial-duel (rush-hour-arena, kill-reactive-enemy-within -ticks-with-loss-cap); duel's easy→hard ladder + easy/medium configs already cover even→outnumbered→entrenched. adversarial-duel is the canonical adversarial scenario.
• economy-time-box → quarantined: = economy-force-buildup + a "keep a building standing" clause (same budget-under-clock decision). Kept economy-force-buildup + economy-investment (the latter adds a distinct economy-vs-military allocation choice).

Kept (NOT aimless dup — each serves a purpose):

• strategy-dilemma vs strategy-gauntlet: same objective but different maps (singles-dilemma vs singles-gauntlet) → different route puzzle.
• artofwar-decoy-sacrifice vs -lure-the-tiger: same map but genuinely distinct credit-assignment (sacrifice-and-lose vs displace-and-exploit).

Active set: 21 distinct decisions. Quarantined packs remain on disk, runnable via explicit --packs, excluded from the default sweep.

Per-scenario closer look — #1 action-multiunit-coordination

Difficulty axis (one new controlled variable per tier, so a tier failure attributes to a single capability):

• easy — 2 regions, exact (x,y). Tests parallel dispatch: serialized one-group-at-a-time touring blows the binding tick deadline (30+ turns); only genuine simultaneous control (17–20 turns) makes it. Split enforced by units_in_region_gte n=2 (not reach_region, which one touring unit satisfies).
• medium — 3 dispersed regions (NE / SW bottom-left / SE), exact (x,y), attrition cap, infantry pickets on the two eastern lanes. Adds multi-vector dispatch: a single eastward column cannot reach the SW region.
• hard — byte-identical setup to medium (same regions, pickets, deadline, attrition); the ONLY changed variable is objective_coords: relative — the briefing gives compass labels ("the NORTH-EAST corner") instead of coordinates. Tests spatial grounding: the model must localize each target on the minimap (each region holds one enemy building as the on-map landmark). Deliberately NOT spawn-varied and NO turrets on the path — introducing either would add a second uncontrolled variable and break the clean medium→hard attribution (recorded as a reasoned exception to the task-#23 spawn-variation contract in tests/test_hard_tier.py::NOT_APPLICABLE).

objective_coords (exact|relative, schema Level/ScenarioConfig, default exact) is a reusable knob: any pack can opt into coordinate-blind objectives. The win predicate always evaluates against the real engine coordinates regardless of disclosure.

Per-scenario closer look — #2 action-sequenced-execution

Original defect: the pack claimed to test ordered, non-stalling route execution but the win condition referenced only the final region + a tick band — a beeline-to-final that skipped every waypoint and idled to the gate won (verified: model arrived tick 1353, idled ~900 ticks, won). Not model compensation: the scenario didn't enforce what it advertised.

Fix — new reusable stateful predicate waypoint_sequence: latches ordered region-visit progress (monotonic) on a per-episode signals.seq_progress scratch. Wk+1 only counts after Wk; skip / out-of-order / idle ⇒ never satisfied. Coords-aware phrase (exact|relative). Difficulty axis (one new controlled variable per tier):

• easy — ONE ordered 3-waypoint route, clear lanes, generous budget. Run: WIN, W1→W2→W3 in order, tick 1893 < 2400.
• medium — TWO ordered routes that cross, BOTH required ⇒ must split into two parallel columns (serial overruns); attrition cap. Run: LOSS (valid) — model split correctly, finished route S in order, lost column discipline on N and timed out (the exact re-deliberation failure the pack targets).
• hard — TWO 4-waypoint serpentine routes on the larger scout-arena (176×80, scripts/build_scout_arena_map.py), given ONLY by relative search bands (no coords); fogged markers revealed by the enemy_building_spotted interrupt; seed-varied 2 spawn points. Run: LOSS (valid) — machinery all verified working (interrupts fired on the big map); model drove to literal corners instead of scanning the bands, ignored the revealed positions, attrition-bust. Solvable in principle; not compensated.

All tick budgets aligned to ~90 ticks/turn so timeout / attrition / wipe are reachable losses within max_turns (no draw degeneracy). Footgun recorded: a base_map placed at the Level level is silently ignored — it must go inside overrides: (guard test added).

Win-speed bonus (cross-cutting scoring)

Every episode records win_tick / win_turns / win_budget / speed / composite_base (score.json + manifest). On a win the composite gets +SPEED_BONUS·speed (SPEED_BONUS=0.05, speed = 1 − win_tick/budget, budget = tightest within_ticks in the win tree else max_ticks). Bounded so a fast win ranks above a slow win but never lifts a loss above a win or overrides correctness. Leaderboard carries win_speed/win_turns and breaks composite/ win-rate ties by faster wins.

YAML-referenceable generators (mapgen / botgen)

A pack no longer needs a static .oramap + static enemy actors:

• Map — base_map: {generator: arena, name: scout-arena, width: 176, height: 80, cordon: 4} materializes a deterministic, content-addressed .oramap at compile time (openra_bench/mapgen.py; static string ids still work). #2 hard uses this.
• Bot — enemy: {faction: soviet, bot_type: hunt} (alias bot:) attaches an engine-side scripted opponent. Behaviours (map-agnostic, ground-truth, per-tick): hunt (each unit attacks nearest agent unit), rusher (concentrated relentless charge), patrol (oscillate around own spawn, engage intruders), turtle (hold spawn, return fire). Runs in openra-sim/scripted_bot because the Python boundary can't see fogged enemy ids or command the enemy player; the bench surface (openra_bench/botgen.py) validates the name at compile.

Whole-suite no-cheat pass (#1–#21, all 21 active packs)

Every active pack has been individually closer-looked + redesigned to the "no defect, no cheat" bar: a scripted brute / stall / greedy / shortest-path / sweep / wrong-route policy must LOSE on every level + every hard seed; the intended capability policy must WIN; non-win is a real reachable timeout LOSS, not a draw. Engine smoke clean across seeds 1–4 each level (no panics, in-bounds, map_supported); hard tiers in tests/test_hard_tier.py::UPGRADED keep ≥2 distinct seed- driven spawns; reasoned NOT_APPLICABLE exceptions documented in the test. Suite 414 passed, 1 skipped, 0 failed on the consolidated main after all 21 redesigns landed. Author-side artifacts: the SCENARIO_REVIEW_CHECKLIST.md is the methodology; per-scenario commit messages on main document each capability, the cheat path that was closed, and the scripted before/after evidence.

Recurring defect classes the pass eliminated:

• Inert deadlines (within_ticks ≫ tick reachable at max_turns, i.e. > 93+90·(max_turns-1)) ⇒ draw degeneracy. Every level now satisfies within_ticks AND after_ticks ≤ that bound.
• Missing fail_condition ⇒ non-win = draw. Every level now has an any_of fail covering at least the timeout (plus force-loss / attrition where applicable).
• Capability not enforced by the win predicate (the laziest play satisfies it without doing the advertised skill). Fixed with the right predicate per pack: units_in_region_gte (split / main body), waypoint_sequence (ordered / latched), units_lost_lte (bait budget), buildings_discovered_gte (fog-dependent discovery), enemy_key_buildings_destroyed (faithful "destroy fact+proc"), exact-cost starting_cash (binding allocation), etc.
• Off-map actors (engine panic). Every level's actors verified in playable bounds across seeds 1–4.
• Engine auto-done on enemy-elimination terminating before the intended win/fail evaluated → drew. Mitigation: an unarmed high-HP enemy fact marker at the objective ensures the episode runs to a real win or timeout LOSS.

Engine footguns surfaced during the pass (flagged for follow-up):

• power_surplus_gte is currently inert: the obs reports power_provided/power_drained = 0 even with placed powr, so the predicate is effectively 0 ≥ N and never fires meaningfully. Affected packs (economy-investment, tech-production-planning, etc.) intentionally do NOT rely on it as the sole discriminator.
• deploy is not executed by the installed wheel: an MCV reaches the target cell but never becomes a Construction Yard. building- and-planning hard was redesigned around build-radius creep instead.
• Both are tracked as engine-side follow-ups (separate from the pre-existing pending task #11 sabotage/specials).

Note: capture_actor is deliberately absent until the engine gains a Capture order (pending S8 / task #11) — the bench never advertises a tool the engine can't execute (1:1 parity).