yxc20098 commited on
Commit
17f024b
ยท
1 Parent(s): 1400f05

schema: add meta.benchmark_anchor (list[str]); + SCENARIO_BACKLOG.md (200-pack catalog)

Browse files

- ScenarioMeta gains optional benchmark_anchor: list[str] so every
new pack declares the real-world capabilities and/or named
benchmarks it transfers to (e.g. ['MicroRTS rush-defense',
'SC2LE defend-the-cheese tempo']). Field defaults to [] for
backward compatibility with the 22 existing packs (backfill TBD).
- SCENARIO_BACKLOG.md is the canonical 200-pack catalog organized
into 15 families (A-O: base building, multi-front, defense
positioning, combat, scouting, economy, long-horizon, adversarial,
procedural, robustness, special abilities, transport, power,
coordination, time-pressure). Each seed has its real-world or
benchmark anchor + predicate sketch + engineering-cost tag, plus
a prioritized engineering shopping list of the PRs that unblock
the most seeds (forbidden_tools / raider / then-composite /
own_building_lost_gte / etc.).

SCENARIO_BACKLOG.md ADDED
@@ -0,0 +1,317 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # OpenRA-Bench โ€” Scenario Backlog (toward 200 packs)
2
+
3
+ This file is the canonical brainstorm catalog for the path from
4
+ **29 packs today** (22 active + 17 designed in the plan + 2 harvest
5
+ rebuilds) โ†’ **~200 packs total**. It complements
6
+ `SCENARIO_REVIEW_CHECKLIST.md` (methodology) and `SCENARIO_QUALITY.md`
7
+ (whole-suite recap) by giving the next-buildable seed any coding agent
8
+ can pull off.
9
+
10
+ ## How to use this file
11
+
12
+ 1. Pick a seed whose **engineering tag** matches what's available
13
+ today (`YAML` = build now; `BOT:x` / `PRED:x` / `ENG:x` = blocked
14
+ on that engine PR landing).
15
+ 2. Author the pack to the **no defect, no cheat** bar (see
16
+ `SCENARIO_REVIEW_CHECKLIST.md` and `CLAUDE.md`).
17
+ 3. Populate the new schema field `meta.benchmark_anchor: list[str]`
18
+ with the seed's anchor (real-world capability and/or named
19
+ benchmark).
20
+ 4. Scripted-policy validate (stall / brute / wrong-path / intended)
21
+ across seeds 1โ€“4 โ€” every lazy policy must LOSS, intended must WIN,
22
+ never a DRAW.
23
+ 5. Move the seed from this file to a section of
24
+ `SCENARIO_QUALITY.md` (the whole-suite catalog) once committed.
25
+
26
+ ## Engineering tags
27
+
28
+ - `YAML` โ€” buildable today, pure YAML.
29
+ - `MAP:x` โ€” needs a new map generator or static `.oramap`.
30
+ - `BOT:x` โ€” needs new scripted-bot variant in
31
+ `openra-sim/src/scripted_bot.rs`. Currently supported: `hunt`,
32
+ `rusher`, `patrol`, `turtle`, `guard`. Proposed: `raider`, `pincer`,
33
+ `feinter`, `prober`, `tower-rusher`, `boomer`, `tech-switcher`.
34
+ - `PRED:x` โ€” needs new predicate in
35
+ `openra_bench/scenarios/win_conditions.py` (+ `_PHRASES` entry in
36
+ `game_knowledge.py`).
37
+ - `ENG:x` โ€” needs deeper engine work (new order, signal, mid-episode
38
+ hook). Examples: `deploy`, `capture_actor`, `apc-transport-complete`,
39
+ `air-units`, `power-signals` (today inert), `scheduled-event`,
40
+ `dynamic-deadline`, `RL-adversary` (P3-1).
41
+
42
+ ## Engineering shopping list (PRs that unblock the most seeds)
43
+
44
+ Order from cheapest to most expensive; numbers in parentheses are how
45
+ many seeds each PR unblocks.
46
+
47
+ 1. **`forbidden_tools` + `tool_violation` signal** (P2-1) โ€” unblocks
48
+ ~6 seeds (Group I + B3).
49
+ 2. **`raider` bot** โ€” unblocks ~4 (B1 + Group F defenses + C2).
50
+ 3. **`then:[A,B]` happened-before composite** (P2-3) โ€” unblocks ~5
51
+ (B2 + Group G/I/N).
52
+ 4. **`own_building_lost_gte` + `building_destroyed_gte`** (P2-2) โ€”
53
+ unblocks ~3 (C3 + C2 + C5).
54
+ 5. **`region_held_for_ticks`** โ€” strengthens B4 + several defense
55
+ seeds.
56
+ 6. **`pincer` / `feinter` bots** โ€” unblocks C3, C4, several adv-*.
57
+ 7. **Mid-episode scripted-event hook** โ€” unblocks C5 + 4 robustness
58
+ seeds + surprise-flank.
59
+ 8. **Power-signals fix** โ€” unblocks 7+ seeds in Groups A/M.
60
+ 9. **MCV `deploy` order** โ€” unblocks ALL Group A `mcv-*` (7+ seeds).
61
+ 10. **`capture_actor` order** โ€” unblocks Group K spec-engineer-*.
62
+ 11. **Engineer-rebuild order** โ€” unblocks several robustness seeds.
63
+ 12. **`apc` transport completeness** โ€” unblocks Group L.
64
+ 13. **Water mapgen + `lst`** โ€” unblocks Group L amphibious + island.
65
+ 14. **Air units (`yak`/`mig`/`heli`)** โ€” unblocks anti-air,
66
+ paradrop, air-ground coord.
67
+ 15. **Stealth/`spy`/`thief` mechanics** โ€” unblocks spec-spy / spec-thief.
68
+ 16. **Superweapons** (nuke / iron-curtain / chronosphere) โ€” Group K.
69
+ 17. **Live RL adversary slot (P3-1)** โ€” unblocks C1, C4, all
70
+ adv-*-race seeds.
71
+
72
+ ---
73
+
74
+ # Backlog seeds organized by family
75
+
76
+ Each seed has: `id` โ€” cell โ€” real-world anchor // benchmark anchor โ€”
77
+ one-line predicate sketch โ€” engineering tag(s).
78
+
79
+ ## Group A โ€” Base building & MCV (25)
80
+
81
+ Largely engine-gated on `deploy`; pre-`deploy` proxies via pre-placed
82
+ `fact` markers + build-radius creep.
83
+
84
+ - `mcv-deploy-and-build` โ€” Opening MCVโ†’fact site choice // critical-infra siting; SC2LE BO โ€” `has_building:fact` AND deploy-tick โ‰ค T AND `building_total_gte:3` โ€” ENG:deploy + YAML
85
+ - `mcv-deploy-defensible-site` โ€” Opening site under threat // resilience-first siting; MicroRTS terrain-aware โ€” `building_in_region:{good_terrain,fact,1}` AND survival โ€” ENG:deploy + YAML
86
+ - `mcv-deploy-near-resource` โ€” Opening site near ore // econ-first siting; SC2LE econ โ€” `building_in_region:{near_ore,fact,1}` AND `economy_value_gte:N` โ€” ENG:deploy + YAML
87
+ - `mcv-deploy-relocate-under-pressure` โ€” Mid conyard pack + relocate // BCP relocation; PlanBench replanning โ€” `building_in_region:{new_site,fact,1}` after `after_ticks:T_loss` โ€” ENG:deploy+undeploy + YAML
88
+ - `mcv-deploy-second-base` โ€” Mid 2nd MCV โ†’ 2nd base // multi-site ramp; MicroRTS expansion โ€” 2 facts in 2 regions โ€” ENG:deploy + YAML
89
+ - `mcv-deploy-third-base` โ€” Mid-Late 3rd MCV โ†’ 3rd base // tri-base macro; SC2 3-base โ€” 3 facts in 3 regions โ€” ENG:deploy + YAML
90
+ - `mcv-deploy-evac-and-resite` โ€” Mid evacuate failing base // disaster recovery; ScienceWorld โ€” fact razed THEN new fact built โ€” ENG:deploy + YAML
91
+ - `build-radius-creep-to-objective` โ€” Mid walk base toward enemy // FOB; SC2 forward pylon โ€” `building_in_region:{near_enemy,pbox,1}` โ€” YAML
92
+ - `build-radius-creep-bridge-island` โ€” Mid-Late creep over bridge // staging expansion; MicroRTS terrain โ€” `building_in_region:{island,fact,1}` โ€” ENG:bridge-aware-mapgen + YAML
93
+ - `build-power-online-first` โ€” Opening `has_building:powr` before all else // grid bring-up sequencing; PlanBench ordering โ€” `then:[powr, _]` โ€” PRED:then + YAML
94
+ - `build-defensive-tower-line` โ€” Mid 4 pbox in line covering choke // perimeter design; ERQA spatial โ€” `building_count_gte:{type:pbox,n:4}` + `building_in_region:{choke,pbox,4}` โ€” YAML
95
+ - `build-defensive-tower-cluster` โ€” Mid pbox cluster at bottleneck // concentrated defense; ERQA โ€” `building_in_region:{bottleneck,pbox,4}` โ€” YAML
96
+ - `build-defensive-skirt-corners` โ€” Mid corner-mounted defenses // distributed defense; MicroRTS โ€” `building_in_region` 4x corner regions โ€” YAML
97
+ - `build-spacing-anti-splash` โ€” Mid space buildings to survive splash // resilience design; robustness โ€” `min_separation_gte:N` โ€” PRED:min_separation + YAML
98
+ - `build-anti-air-positioning` โ€” Mid SAM coverage // anti-air planning; SC2 antiair โ€” `building_in_region` of `sam` โ€” ENG:air-units + YAML
99
+ - `build-engineer-rebuild-after-loss` โ€” Mid engineer rebuilds razed building // resilience drill; PlanBench replanning โ€” building re-presence after loss โ€” ENG:engineer-rebuild-order + YAML
100
+ - `build-sequence-tech-fastest` โ€” Opening BO hits `weap` in min turns // BOM-optimal; PlanBench cost-optimal โ€” `then:[tent, weap]` AND `within_ticks:T_tight` โ€” PRED:then + YAML
101
+ - `build-sequence-tech-cheapest` โ€” Opening BO hits `weap` cheapest // BOM-cost; PlanBench cost-optimal โ€” cash-bound โ€” YAML
102
+ - `build-sequence-tech-most-resilient` โ€” Mid tech via redundant prereqs // resilience; PlanBench robust โ€” 2x prereq buildings โ€” PRED:then + YAML
103
+ - `build-power-budget-online` โ€” Opening keep `power_surplus_gte:0` // grid bring-up; SC2 power โ€” ENG:power-signals + YAML
104
+ - `build-power-down-defensive` โ€” Mid `power_down` non-essentials under attack // load-shedding; ops runbook โ€” YAML
105
+ - `build-repair-priority-under-fire` โ€” Mid repair structures in correct order // repair triage; DR โ€” YAML
106
+ - `build-sell-and-rebuild-elsewhere` โ€” Mid `sell` weakened, rebuild defended // capital reallocation; finance โ€” YAML
107
+ - `build-rally-point-management` โ€” Opening set rally forward // production logistics; warehouse SLA โ€” YAML
108
+ - `build-production-throughput-multibuilding` โ€” Mid 2x `weap` parallel queue // throughput; queueing theory โ€” YAML
109
+
110
+ ## Group B โ€” Multi-front base building (12)
111
+
112
+ - `mfb-two-base-simultaneous` โ€” Mid build base-2 while base-1 builds // multi-site ramp; SC2 sim-base โ€” 2 facts within tight window โ€” YAML
113
+ - `mfb-base-1-defend-base-2-build` โ€” Mid defend live + build forward // ops continuity during expansion; DR siting โ€” YAML
114
+ - `mfb-third-base-against-clock` โ€” Mid-Late 3rd base before deadline // greedy expansion; SC2 3-base timing โ€” YAML
115
+ - `mfb-relocate-conyard-under-pressure` โ€” Mid pack + move conyard // BCP; PlanBench replanning โ€” ENG:deploy/undeploy + YAML
116
+ - `mfb-evacuate-base-deploy-elsewhere` โ€” Mid abandon base, redeploy MCV // disaster recovery; ScienceWorld โ€” ENG:deploy + YAML
117
+ - `mfb-tech-base-vs-economy-base` โ€” Mid-Late role-specialized bases // specialization; industrial โ€” YAML
118
+ - `mfb-island-base-amphibious` โ€” Late build base across water // amphibious staging; RoboMaster ship-to-shore โ€” ENG:transport+amphib-mapgen + YAML
119
+ - `mfb-forward-base-near-enemy-corner` โ€” Mid-Late FOB // SC2 forward production โ€” YAML
120
+ - `mfb-redundant-tech-buildings` โ€” Mid 2x `weap` for resilience // redundancy; robust planning โ€” YAML
121
+ - `mfb-mirror-base-east-west` โ€” Mid symmetric multi-base // distributed-systems mirror; raft replicas โ€” YAML
122
+ - `mfb-rotating-production-pressure` โ€” Mid-Late alternate production under raid // load balancing; queueing โ€” PRED:then + YAML
123
+ - `mfb-supply-line-link-between-bases` โ€” Mid-Late connect 2 bases via patrol // logistics route; supply-chain โ€” YAML
124
+
125
+ ## Group C โ€” Defense positioning & surprise reaction (16)
126
+
127
+ - `def-position-expected-direction` โ€” Mid fortify N (intel says N) // pre-positioned defense; ERQA spatial commit โ€” YAML
128
+ - `def-position-revealed-direction` โ€” Mid fortify after scout reveals // adaptive defense; PlanBench replanning โ€” YAML
129
+ - `def-surprise-flank-react` โ€” Mid intel said N, enemy comes S // surprise reaction; adversarial robustness โ€” ENG:enemy-spawn-misdirection + YAML
130
+ - `def-pre-position-mobile-reserve` โ€” Mid mobile reserve in center // reserve doctrine; military โ€” YAML
131
+ - `def-tower-line-vs-cluster` โ€” Mid choose spread vs concentration // defense topology; graph min-cut โ€” YAML
132
+ - `def-walls-vs-towers` โ€” Mid passive vs active mitigation // security architecture โ€” ENG:walls + YAML
133
+ - `def-engineer-repair-under-fire` โ€” Mid which tower to repair under fire // triage; DR โ€” YAML
134
+ - `def-stance-mgmt-hold-then-attack` โ€” Mid hold-fireโ†’return-fire timing // ROE drill โ€” YAML
135
+ - `def-while-building` โ€” Mid defend with workforce while building defenses // ops continuity โ€” YAML
136
+ - `def-multi-direction` โ€” Mid defend 3 lanes // distributed defense; graph min-cut โ€” YAML
137
+ - `def-in-depth` โ€” Mid-Late front + 2nd line vs single thick // defense-in-depth; security doctrine โ€” YAML
138
+ - `def-with-ambush` โ€” Mid fog ambush vs static // active deception; SC2 hidden โ€” YAML
139
+ - `def-evacuation` โ€” Mid-Late evacuate overwhelmed position // BCP; emergency mgmt โ€” YAML
140
+ - `def-retreat-and-rebuild` โ€” Mid-Late concede + rebuild deeper // strategic withdrawal; CICERO retreat โ€” YAML
141
+ - `def-anti-air-positioning` โ€” Late SAM placement // anti-air; SC2 โ€” ENG:air-units + YAML
142
+ - `def-tower-rush-counter` โ€” Opening enemy tower-rushes you // tower-rush defense; SC2 cannon-rush counter โ€” BOT:tower-rusher + YAML
143
+
144
+ ## Group D โ€” Combat micro & tactics (20)
145
+
146
+ - `combat-focus-fire-priority` โ€” Mid which enemy to kill first // target prioritization; SC2 focus-fire โ€” YAML
147
+ - `combat-kite-jeep-vs-tank` โ€” Mid kite ranged // micro; SC2 kiting โ€” YAML
148
+ - `combat-stance-mgmt-attack` โ€” Mid switch to attack-anything // ROE; SC2 stance โ€” YAML
149
+ - `combat-formation-tank-wedge` โ€” Mid tank wedge // formation; military โ€” YAML
150
+ - `combat-formation-infantry-screen-tanks` โ€” Mid infantry screens armor // combined-arms; military โ€” YAML
151
+ - `combat-retreat-after-engagement` โ€” Mid disengage timing // tactical withdrawal; SC2 โ€” YAML
152
+ - `combat-flanking-attack` โ€” Mid flank vs head-on // flank maneuver; military โ€” YAML
153
+ - `combat-pincer-coordination` โ€” Mid two-prong // pincer; SC2 multi-prong โ€” YAML
154
+ - `combat-bait-counter-attack` โ€” Mid lure + flank-counter // bait; SC2 bait โ€” BOT:guard + YAML (idiom proven)
155
+ - `combat-divide-and-conquer` โ€” Mid split enemy via feint // CICERO โ€” YAML
156
+ - `combat-protect-vip-escort` โ€” Mid escort VIP // diplomatic escort โ€” PRED:`unit_type_count_gte:{vip,1}`-survival + YAML
157
+ - `combat-suicide-charge-mission` โ€” Mid sacrifice everything for X // forlorn hope; military โ€” YAML
158
+ - `combat-prevent-retreat` โ€” Mid cut off enemy retreat // encirclement; military โ€” YAML
159
+ - `combat-defensive-perimeter-around-vip` โ€” Mid perimeter // VIP cordon โ€” YAML
160
+ - `combat-skirmish-then-disengage` โ€” Mid bleed and pull back // skirmisher; SC2 โ€” YAML
161
+ - `combat-vehicle-vs-infantry-counter` โ€” Mid tank vs inf // RPS counter; SC2 hard counter โ€” YAML
162
+ - `combat-rocket-soldier-anti-vehicle` โ€” Mid e3 vs tank // RPS; SC2 โ€” YAML
163
+ - `combat-tank-vs-tank-engagement` โ€” Mid mirror combat // SC2 mirror micro โ€” YAML
164
+ - `combat-amphibious-landing` โ€” Mid-Late APC across water // amphibious assault; D-Day โ€” ENG:transport+water-mapgen + YAML
165
+ - `combat-attack-from-behind-fog` โ€” Mid emerge from fog // surprise attack; SC2 hidden โ€” YAML
166
+
167
+ ## Group E โ€” Scouting / Perception (15)
168
+
169
+ - `scout-far-frontier` โ€” Early long-distance scout // recon; ERQA โ€” YAML
170
+ - `scout-and-survive` โ€” Early scout returns alive // scout-preserve; SC2 worker scout โ€” YAML
171
+ - `scout-multiple-fog-areas` โ€” Early parallel scout // Watch-And-Help โ€” YAML
172
+ - `scout-detect-enemy-tech` โ€” Early-Mid identify enemy via buildings // tech-read; PlanBench obs โ€” YAML
173
+ - `scout-detect-base-direction` โ€” Early which corner has enemy // direction-read; ERQA โ€” YAML
174
+ - `scout-count-defenders` โ€” Early-Mid exact defender count // census; POMDP โ€” YAML
175
+ - `scout-discover-hidden-base` โ€” Mid off-axis hidden base // deep recon; CICERO โ€” YAML
176
+ - `scout-detect-incoming-army` โ€” Mid early-warn // intrusion detection โ€” YAML
177
+ - `scout-detect-air-threat` โ€” Late air detection // anti-air recon โ€” ENG:air-units + YAML
178
+ - `scout-detect-stealth-units` โ€” Late stealth detection // SC2 detector โ€” ENG:stealth-units + YAML
179
+ - `scout-map-reveal-percent-target` โ€” Early reveal N% // coverage; ERQA โ€” YAML
180
+ - `scout-and-report` โ€” Early-Mid reveal then return // recon-report cycle; mil intel โ€” YAML
181
+ - `scout-jeep-vs-infantry-cost-effective` โ€” Early cheap fast scout // econ scout; SC2 worker-vs-Reaper โ€” YAML
182
+ - `scout-cycle-keep-info-fresh` โ€” Mid continuous cadence // info staleness; intel ops โ€” PRED:then-repeated + YAML
183
+ - `scout-info-blackout-react` โ€” Mid fog blanket then enemy revealed // dynamic perception; adversarial โ€” ENG:fog-shift-hook + YAML
184
+
185
+ ## Group F โ€” Economy / Resources (16)
186
+
187
+ Mostly unblocked by S0/S1 (task #14).
188
+
189
+ - `econ-startup-from-scratch` โ€” Opening first proc + harv // greenfield econ; SC2 first base โ€” YAML
190
+ - `econ-harvester-pathing-optimization` โ€” Opening harv route choice // logistics; queueing โ€” PRED:`econ_rate_gte` + YAML
191
+ - `econ-multi-patch-allocation` โ€” Mid which ore patches // resource allocation; OR โ€” YAML
192
+ - `econ-far-patch-vs-near-patch` โ€” Mid travel vs throughput // econ tradeoff; OR โ€” YAML
193
+ - `econ-silo-vs-spend` โ€” Mid store or spend // capital allocation; PlanBench โ€” YAML
194
+ - `econ-replace-dead-harvester` โ€” Mid rebuild dead harv // resilience; PlanBench replanning โ€” YAML
195
+ - `econ-protect-harvester-route` โ€” Mid defend harv path // econ defense; SC2 harass defense โ€” YAML
196
+ - `econ-cash-reserve-management` โ€” Mid never zero cash // financial reserve โ€” PRED:`cash_gte` + YAML
197
+ - `econ-target-cash-amount-by-deadline` โ€” Mid cash bar by deadline // budget by date; finance โ€” YAML
198
+ - `econ-contention-with-enemy` โ€” Mid shared ore patch // contested resource; game theory โ€” YAML
199
+ - `econ-recover-from-zero-cash` โ€” Mid start from 0 cash // bankruptcy comeback โ€” YAML
200
+ - `econ-quantitative-vs-qualitative-spend` โ€” Mid many cheap vs few expensive // unit composition; OR โ€” YAML
201
+ - `econ-burn-rate-management` โ€” Mid spend at target rate // runway mgmt; startup finance โ€” PRED:`spend_rate_lte` + YAML
202
+ - `econ-buy-vs-build-decision` โ€” Mid cash now or save for tech // PlanBench โ€” YAML
203
+ - `econ-resource-trade-with-self` โ€” Mid-Late balance ore vs cash // commodity hedging; finance โ€” YAML
204
+ - `econ-overflow-to-silos` โ€” Mid silo-required throughput // capacity planning; backpressure โ€” YAML
205
+
206
+ ## Group G โ€” Long-horizon / Multi-phase (12)
207
+
208
+ - `lh-opening-to-tech-to-army` โ€” Late 4-phase chain // long-horizon; SC2 macro โ€” YAML
209
+ - `lh-opening-to-defense-to-counter` โ€” Late defense โ†’ counter // counter-attack timing; SC2 timing โ€” YAML
210
+ - `lh-tech-pivot-attack` โ€” Late tech โ†’ pivot โ†’ attack // adaptive macro; PlanBench โ€” YAML
211
+ - `lh-econ-army-victory` โ€” Late econ โ†’ army โ†’ victory // SC2 macro โ€” YAML
212
+ - `lh-defense-tech-second-base` โ€” Late defend โ†’ tech โ†’ expand // SC2 secure-expand โ€” YAML
213
+ - `lh-scout-react-counter` โ€” Late scout โ†’ react โ†’ counter // CICERO โ€” YAML
214
+ - `lh-100-turn-marathon-survival` โ€” Very-Late survive 100 turns // lmgame-Bench multi-hour โ€” YAML (max_turns:100+)
215
+ - `lh-build-army-coordinate-multifront-attack` โ€” Late assemble + coordinate // operational planning; military โ€” YAML
216
+ - `lh-progression-stage-locked` โ€” Late each stage gates next // PERT; PlanBench โ€” PRED:then + YAML
217
+ - `lh-credit-only-final-phase` โ€” Late sparse reward // long-horizon RL โ€” YAML
218
+ - `lh-multi-checkpoint-5-plus` โ€” Late 5+ checkpoints // sequenced ops; PlanBench โ€” PRED:waypoint_sequence-extended + YAML
219
+ - `lh-recovery-after-mid-game-loss` โ€” Late comeback // SC2 comeback โ€” ENG:scheduled-event + YAML
220
+
221
+ ## Group H โ€” Adversarial / Game-theoretic (12)
222
+
223
+ - `adv-counter-strategy-read` (= C1 from plan)
224
+ - `adv-feint-handling` (= C4 from plan)
225
+ - `adv-bluff-detection` โ€” Mid commit vs hesitate // poker-bench โ€” ENG:RL-adversary + YAML
226
+ - `adv-rps-counter-pick` โ€” Mid unit-counter selection // RPS; game theory โ€” YAML
227
+ - `adv-tempo-vs-adversary` โ€” Mid-Late out-tempo // TextStarCraft II โ€” ENG:RL-adversary + YAML
228
+ - `adv-economy-race` โ€” Late out-econ // SC2 econ race โ€” ENG:RL-adversary + YAML
229
+ - `adv-army-race` โ€” Late out-army // SC2 army race โ€” ENG:RL-adversary + YAML
230
+ - `adv-tech-race` โ€” Late out-tech // SC2 tech race โ€” ENG:RL-adversary + YAML
231
+ - `adv-base-trade-race` (= C3 from plan)
232
+ - `adv-asymmetric-weaker-must-win` โ€” Mid outnumbered must-win // underdog; RTS asymmetric โ€” YAML
233
+ - `adv-asymmetric-stronger-no-loss` โ€” Mid dominate without loss // overdog discipline; CICERO โ€” YAML
234
+ - `adv-mirror-match-best-strategy` โ€” Late mirror best-of // SC2 mirror โ€” ENG:RL-adversary + YAML
235
+
236
+ ## Group I โ€” Procedural / Tool-use compliance (10)
237
+
238
+ - `proc-strict-toolban-fidelity` โ€” Cross BFCL relevance // BFCL V4 โ€” PRED:forbidden_tools + YAML
239
+ - `proc-strict-toolban-under-pressure` (= B3 from plan)
240
+ - `proc-checklist-no-deviation` โ€” Cross exact ordered checklist // SOP compliance; IFBench โ€” PRED:waypoint_sequence strict + YAML
241
+ - `proc-ordered-action-strict` โ€” Cross exact action sequence // PlanBench strict ordering โ€” PRED:then + YAML
242
+ - `proc-no-attack-passive-only` โ€” Cross only move+stop // ROE; mil โ€” PRED:forbidden_tools + YAML
243
+ - `proc-only-build-no-combat` โ€” Cross only build/place // role-based access โ€” PRED:forbidden_tools + YAML
244
+ - `proc-only-defend-no-attack` โ€” Cross only defensive moves // ROE; mil โ€” PRED:forbidden_tools + YAML
245
+ - `proc-tool-use-with-distractor` โ€” Cross ignore distracting tool // ฯ„ยฒ-bench โ€” YAML
246
+ - `proc-tool-use-multi-distractor` โ€” Cross ignore multiple distractors // ฯ„ยฒ-bench โ€” YAML
247
+ - `proc-instruction-following-edge-case` โ€” Cross instruction-edge // IFBench โ€” YAML
248
+
249
+ ## Group J โ€” Robustness / Edge cases (8)
250
+
251
+ - `rob-unit-loss-recovery` โ€” Mid recover from N losses // resilience; PlanBench replanning โ€” YAML
252
+ - `rob-building-loss-rebuild` (= C5 from plan)
253
+ - `rob-cash-depletion-recovery` โ€” Mid recover from 0 cash // bankruptcy โ€” YAML
254
+ - `rob-power-failure-recovery` โ€” Mid recover from brownout // grid restoration; DR โ€” ENG:power-signals + YAML
255
+ - `rob-multiple-simultaneous-pressures` โ€” Mid econ + def + tech under pressure // overwhelmed ops; triage โ€” YAML
256
+ - `rob-unexpected-enemy-spawn` โ€” Mid spawn mid-episode // surprise; adversarial robustness โ€” ENG:scheduled-spawn + YAML
257
+ - `rob-objective-change-midway` โ€” Mid win-condition shifts at T // goal-conditional RL โ€” ENG:objective-swap-hook + YAML
258
+ - `rob-deadline-shortened-midway` โ€” Mid deadline brought forward // schedule compression; project mgmt โ€” ENG:dynamic-deadline-hook + YAML
259
+
260
+ ## Group K โ€” Special abilities (engine-gated, 10)
261
+
262
+ - `spec-engineer-capture` โ€” Mid capture enemy `proc` // industrial espionage; SC2 spy steal โ€” ENG:capture_actor + YAML
263
+ - `spec-engineer-mass-capture` โ€” Mid capture all of enemy tech // takeover; CICERO โ€” ENG:capture_actor + YAML
264
+ - `spec-spy-infiltrate` โ€” Mid reveal enemy tech via spy // intel infiltration; SC2 scan โ€” ENG:spy + YAML
265
+ - `spec-thief-steal-resources` โ€” Mid steal enemy cash // econ sabotage; SC2 worker raid โ€” ENG:thief + YAML
266
+ - `spec-medic-heal-coordination` โ€” Mid medic heals infantry // SC2 medivac โ€” ENG:medic + YAML
267
+ - `spec-airborne-paradrop` โ€” Late drop reinforcements // SC2 paradrop โ€” ENG:airborne + YAML
268
+ - `spec-nuke-strike` โ€” Very-late superweapon // SC2 nuke โ€” ENG:superweapon + YAML
269
+ - `spec-iron-curtain-buff` โ€” Late invuln buff // SC2 cloak โ€” ENG:iron-curtain + YAML
270
+ - `spec-chronosphere-teleport` โ€” Late unit teleport // SC2 warpgate โ€” ENG:chronosphere + YAML
271
+ - `spec-spy-detect-tech` โ€” Mid spy + tech reveal // counterintel; CICERO โ€” ENG:spy + YAML
272
+
273
+ ## Group L โ€” Transport / Amphibious (6)
274
+
275
+ - `transport-apc-deliver-infantry` โ€” Mid APC delivers infantry // last-mile delivery; logistics โ€” ENG:apc-transport + YAML
276
+ - `transport-amphibious-landing` โ€” Mid-Late LST water cross // amphibious; D-Day โ€” ENG:water-mapgen+lst + YAML
277
+ - `transport-evacuate-by-transport` โ€” Mid evac units // BCP evacuation; emergency โ€” ENG:apc-transport + YAML
278
+ - `transport-multiple-trips` โ€” Mid ferry multiple loads // queueing โ€” ENG:apc-transport + YAML
279
+ - `transport-protect-en-route` โ€” Mid escort transport // convoy escort; mil โ€” ENG:apc-transport + YAML
280
+ - `transport-amphibious-island-base` โ€” Mid-Late build on island // amphibious base; strategic geography โ€” ENG:water-mapgen+lst + YAML
281
+
282
+ ## Group M โ€” Power / Maintenance (7)
283
+
284
+ Most are gated on the engine power-signals fix (`power_surplus_gte`
285
+ is currently inert โ€” see CLAUDE.md footguns).
286
+
287
+ - `power-budget-online` โ€” Opening keep `power_surplus_gte:0` // grid bring-up; SC2 power โ€” ENG:power-signals + YAML
288
+ - `power-down-non-essential-under-attack` โ€” Mid load-shed // ops runbook โ€” ENG:power-signals + YAML
289
+ - `power-restoration-order-after-loss` โ€” Mid restore in correct order // utility restoration; DR โ€” ENG:power-signals + YAML
290
+ - `power-tech-vs-defense-conflict` โ€” Mid limited power: tech vs def // capacity allocation; OR โ€” ENG:power-signals + YAML
291
+ - `maint-repair-priority-order` โ€” Mid repair triage // DR โ€” YAML
292
+ - `maint-sell-and-recoup-cash` โ€” Mid sell + recoup // capital reallocation; finance โ€” YAML
293
+ - `maint-rebuild-from-power-failure` โ€” Mid restore after brownout // DR; utility โ€” ENG:power-signals + YAML
294
+
295
+ ## Group N โ€” Coordination & Multi-agent (8)
296
+
297
+ - `coord-squad-handoff` โ€” Mid squad A to squad B handoff // multi-agent handoff; Watch-And-Help โ€” YAML
298
+ - `coord-relay-attack` โ€” Mid A attacks, B follows up // relay; SMAC โ€” YAML
299
+ - `coord-mutual-support` โ€” Mid stay in mutual support range // mil doctrine โ€” PRED:max_inter_unit_distance_lte + YAML
300
+ - `coord-converge-on-target` โ€” Mid 3 squads converge // SMAC โ€” YAML
301
+ - `coord-cover-and-move` โ€” Mid A covers while B moves // fire-and-maneuver; mil โ€” YAML
302
+ - `coord-diversionary-attack` โ€” Mid A diverts, B attacks real target // CICERO โ€” YAML
303
+ - `coord-air-ground-combined` โ€” Late (air) combined-arms // mil โ€” ENG:air-units + YAML
304
+ - `coord-naval-ground-combined` โ€” Late (naval) combined naval-ground // D-Day โ€” ENG:naval + YAML
305
+
306
+ ## Group O โ€” Time-pressure variants (8)
307
+
308
+ (Each is a "+ tight clock" variant of an existing scenario family.)
309
+
310
+ - `tp-rush-objective-very-fast` โ€” Mid speedrun objective // speedrun; human speedrun โ€” YAML
311
+ - `tp-rush-objective-tighter` โ€” Mid tighter speedrun โ€” YAML
312
+ - `tp-rush-multi-objective` โ€” Mid speedrun 2 objectives // speedrun parallel โ€” YAML
313
+ - `tp-survive-N-turns` โ€” Mid survive until clock // survival; SC2 survival โ€” YAML
314
+ - `tp-survive-and-grow` โ€” Mid survive + grow // ops continuity; SRE โ€” YAML
315
+ - `tp-survive-and-strike-at-window` โ€” Mid survive then strike // SC2 timing window โ€” PRED:then + YAML
316
+ - `tp-decision-under-clock` โ€” Mid quick correct choice // tense decision; poker โ€” YAML
317
+ - `tp-pressure-procedural` โ€” Cross procedural under tight clock // procedure under stress; IFBench โ€” PRED:forbidden_tools + YAML
openra_bench/scenarios/schema.py CHANGED
@@ -62,6 +62,19 @@ class ScenarioMeta(BaseModel):
62
  ..., min_length=10, description="Concrete robotics/agentic parallel"
63
  )
64
  author: str = "unknown"
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  # Hygiene (audit): "quarantine" keeps the file on disk and runnable
66
  # by explicit --packs, but excludes it from the default eval set so
67
  # it doesn't dilute scores / the leaderboard. Used for the
 
62
  ..., min_length=10, description="Concrete robotics/agentic parallel"
63
  )
64
  author: str = "unknown"
65
+ # Real-world / benchmark anchors โ€” every pack must name at least one
66
+ # external referent (a named benchmark or a real-world capability)
67
+ # so the scenario carries transfer signal, not just RTS novelty.
68
+ # Multi-anchor packs are common (e.g. tool-fidelity โ†’ BFCL V4 +
69
+ # ฯ„ยฒ-bench + IFBench), so this is a list. The list-non-empty rule
70
+ # is suite-enforced by tests/test_benchmark_anchor_required.py.
71
+ benchmark_anchor: list[str] = Field(
72
+ default_factory=list,
73
+ description="Named benchmarks and/or real-world capabilities this "
74
+ "scenario transfers to (e.g. ['MicroRTS rush-defense', "
75
+ "'SC2LE defend-the-cheese tempo', 'incident-response: defend "
76
+ "production infra under live attack']).",
77
+ )
78
  # Hygiene (audit): "quarantine" keeps the file on disk and runnable
79
  # by explicit --packs, but excludes it from the default eval set so
80
  # it doesn't dilute scores / the leaderboard. Used for the