SCENARIO_REVIEW_CHECKLIST.md

Per-Scenario Closer-Look Checklist

The reproducible methodology applied to #1 (action-multiunit-coordination) and #2 (action-sequenced-execution). Every remaining active pack gets the same treatment: closer look → fix defects → re-verify → run each difficulty easy→medium→hard on the model → inspect the playback → commit/push. Read this fully before touching a pack.

Guiding principle: the benchmark must validly test its stated capability; do NOT compensate for model weakness, and do NOT over-engineer. A model failing a correctly-designed scenario is a good result (real discrimination). Only fix genuine scenario defects.

---

A. SOLVENCY — can the intended strategy actually win, in budget?

1. The win predicate must enforce the advertised capability — and only that. The classic defect: the prose claims X but the win condition is satisfiable without doing X.
   • #1: "split the force" but reach_region (≥1 unit) let one touring unit win → switched to units_in_region_gte (≥2 in EACH region).
   • #2: "ordered route" but only the final region was a predicate → a beeline that skipped every waypoint won → added the stateful waypoint_sequence latch (Wk+1 only counts after Wk; skip / out-of-order / idle ⇒ never satisfied).
   • Ask: "what is the laziest play that satisfies this win condition?" If it isn't the intended capability, the predicate is wrong.
2. Is the optimal/intended play winnable within the tick budget? Estimate path length; engine advances ~90 ticks per decision turn (tick ≈ 93 + 90·(turn-1)). The intended strategy must finish comfortably under within_ticks; the inefficient strategy must NOT (that gap is the discrimination).
3. Coordinate-blind objectives must be solvable. If objective_coords: relative, the model can't cell-count — it needs a feedback loop (an enemy_building_spotted interrupt revealing the marker, a landmark building at each region) and a forgiving enough radius. A "search band" beats a bare compass word.
4. Map fits the actors. Every actor/region coordinate must be inside the map's playable bounds. Actors at scout-arena coords on a rush-hour map → engine panic. Confirm compiled.map_supported and that base_map resolves to the intended map.

B. STABILITY — deterministic, no crashes, fail is reachable

5. Non-win must be a real LOSS, not a draw. Every level needs a fail_condition. Idiom: any_of[ {after_ticks: BUDGET+1}, {not:{units_lost_lte: N}}, {not:{own_units_gte: 1}} ].
6. Tick/turn alignment (critical, easy to get wrong). A fail after_ticks: K only bites if K is reachable within max_turns: require K ≤ 93 + 90·(max_turns-1). Likewise the episode must be able to reach within_ticks before max_turns ends, or a staller draws instead of losing. Re-derive this for every level after any budget/turn change.
7. base_map override goes INSIDE overrides: — a Level-level base_map key is silently ignored (it's not a Level field).
8. Smoke the engine path before a full run: compile the level, _scenario_to_tmp_yaml, RustEnvPool reset(seed) + a few env.step; if the pack enables interrupts, also drive raw_env.step_until_event([...],None,5,[sig]) ~30 steps — catches panics/oob without burning a model run.
9. Hard-tier spawn contract: if the pack is in tests/test_hard_tier.py::UPGRADED, hard must keep ≥2 spawn_point groups (seed-varied start). A deliberate exception is allowed only with a stated reason added to NOT_APPLICABLE.

C. CAPABILITY — clean difficulty axis, faithful framing

10. One new controlled variable per tier so a tier failure attributes to a single capability. easy = the bare skill; medium = +1 axis (a third region / parallel split / contest / attrition); hard = +1 more (relative coords + scouting, larger map, strict budget). Avoid stacking uncontrolled variables.
11. Keep established idioms (e.g. the single-final-region + [after_ticks, within_ticks] band for "execute, don't stall"; fact+proc key-building destruction for adversarial). Don't invent a new mechanism when one exists — but DO add a reusable predicate / engine feature when the capability genuinely can't be expressed (units_in_region_gte, waypoint_sequence, enemy_building_ spotted interrupt, scripted enemy.bot).
12. Title/description are plain and self-explanatory; the objective brief the model sees (game_knowledge.objective_brief) must state the exact machine win/fail in plain language with no degenerate lines.

D. RUN & INSPECT (one difficulty at a time)

13. Run on qwen/qwen3.6-flash via OpenRouter (key in ./.env, git-ignored): python3 -m openra_bench.run_eval --packs openra_bench/scenarios/packs/<pack>.yaml --levels <lvl> --seeds 1 --provider openrouter --model qwen/qwen3.6-flash --playback playback/run1. easy, then medium, then hard.
14. Inspect the playback (playback/run1/*/<pack>:<lvl>:public/ seed1/): manifest.json (outcome/turns), score.json (composite/weakest_link/speed), turns.jsonl (per-turn units, enemies, signals, interrupt, goal), messages.json (model text is in the reasoning field, not content). Reconstruct: did the intended mechanism fire? final positions vs win regions? units_lost? terminal "episode end" frame present?
15. Classify the outcome: scenario defect (→ fix, re-verify, re-run) vs legitimate model failure (→ record as valid discrimination, no change). Cite evidence from the playback.

E. RE-VERIFY & SHIP

16. python3 -m pytest tests/ -q fully green (add/extend focused tests for any new predicate/knob/scenario behaviour).
17. python3 scripts/gen_scenario_docs.py (regenerate the HTML catalog) when prose/objectives change.
18. Commit per fixed scenario, no Claude co-author line, using git -c commit.gpgsign=false commit. Then git fetch -q origin && git rebase -q origin/main && git push -q origin HEAD. Engine changes (OpenRA-Rust) commit+push separately; rebuild the wheel (maturin develop --release) if the engine changed and re-run the affected scenario.
19. Record a "Per-scenario closer look — #N" note in SCENARIO_QUALITY.md (the defect found, the fix, the easy/medium/hard outcomes + verdict).

Reference: defect patterns already seen

• Win condition doesn't enforce the stated capability (laziest play wins). — #1, #2
• reach_region/single-region where a split or ordered visit is intended. — #1, #2
• Missing fail_condition ⇒ non-win == draw. — #1, #2
• after_ticks fail unreachable within max_turns (tick/turn misalignment). — #2
• Relative objective too literal ("NE corner" → map extreme), region inset & unreachable blind, landmark fogged ⇒ unfair. — #1 hard
• base_map at Level level silently ignored. — #2 hard
• Actors placed off the resolved map ⇒ engine panic in the interrupt path. — #2 hard
• Bench advertises a tool the engine can't execute (1:1 parity). — capture_actor / S8
• Inert deadline (within_ticks ≫ optimal) ⇒ no anti-stall teeth. — #1 easy, #2 easy (acceptable for easy only).