Spaces:
Running
schema: add meta.benchmark_anchor (list[str]); + SCENARIO_BACKLOG.md (200-pack catalog)
Browse files- ScenarioMeta gains optional benchmark_anchor: list[str] so every
new pack declares the real-world capabilities and/or named
benchmarks it transfers to (e.g. ['MicroRTS rush-defense',
'SC2LE defend-the-cheese tempo']). Field defaults to [] for
backward compatibility with the 22 existing packs (backfill TBD).
- SCENARIO_BACKLOG.md is the canonical 200-pack catalog organized
into 15 families (A-O: base building, multi-front, defense
positioning, combat, scouting, economy, long-horizon, adversarial,
procedural, robustness, special abilities, transport, power,
coordination, time-pressure). Each seed has its real-world or
benchmark anchor + predicate sketch + engineering-cost tag, plus
a prioritized engineering shopping list of the PRs that unblock
the most seeds (forbidden_tools / raider / then-composite /
own_building_lost_gte / etc.).
- SCENARIO_BACKLOG.md +317 -0
- openra_bench/scenarios/schema.py +13 -0
|
@@ -0,0 +1,317 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# OpenRA-Bench โ Scenario Backlog (toward 200 packs)
|
| 2 |
+
|
| 3 |
+
This file is the canonical brainstorm catalog for the path from
|
| 4 |
+
**29 packs today** (22 active + 17 designed in the plan + 2 harvest
|
| 5 |
+
rebuilds) โ **~200 packs total**. It complements
|
| 6 |
+
`SCENARIO_REVIEW_CHECKLIST.md` (methodology) and `SCENARIO_QUALITY.md`
|
| 7 |
+
(whole-suite recap) by giving the next-buildable seed any coding agent
|
| 8 |
+
can pull off.
|
| 9 |
+
|
| 10 |
+
## How to use this file
|
| 11 |
+
|
| 12 |
+
1. Pick a seed whose **engineering tag** matches what's available
|
| 13 |
+
today (`YAML` = build now; `BOT:x` / `PRED:x` / `ENG:x` = blocked
|
| 14 |
+
on that engine PR landing).
|
| 15 |
+
2. Author the pack to the **no defect, no cheat** bar (see
|
| 16 |
+
`SCENARIO_REVIEW_CHECKLIST.md` and `CLAUDE.md`).
|
| 17 |
+
3. Populate the new schema field `meta.benchmark_anchor: list[str]`
|
| 18 |
+
with the seed's anchor (real-world capability and/or named
|
| 19 |
+
benchmark).
|
| 20 |
+
4. Scripted-policy validate (stall / brute / wrong-path / intended)
|
| 21 |
+
across seeds 1โ4 โ every lazy policy must LOSS, intended must WIN,
|
| 22 |
+
never a DRAW.
|
| 23 |
+
5. Move the seed from this file to a section of
|
| 24 |
+
`SCENARIO_QUALITY.md` (the whole-suite catalog) once committed.
|
| 25 |
+
|
| 26 |
+
## Engineering tags
|
| 27 |
+
|
| 28 |
+
- `YAML` โ buildable today, pure YAML.
|
| 29 |
+
- `MAP:x` โ needs a new map generator or static `.oramap`.
|
| 30 |
+
- `BOT:x` โ needs new scripted-bot variant in
|
| 31 |
+
`openra-sim/src/scripted_bot.rs`. Currently supported: `hunt`,
|
| 32 |
+
`rusher`, `patrol`, `turtle`, `guard`. Proposed: `raider`, `pincer`,
|
| 33 |
+
`feinter`, `prober`, `tower-rusher`, `boomer`, `tech-switcher`.
|
| 34 |
+
- `PRED:x` โ needs new predicate in
|
| 35 |
+
`openra_bench/scenarios/win_conditions.py` (+ `_PHRASES` entry in
|
| 36 |
+
`game_knowledge.py`).
|
| 37 |
+
- `ENG:x` โ needs deeper engine work (new order, signal, mid-episode
|
| 38 |
+
hook). Examples: `deploy`, `capture_actor`, `apc-transport-complete`,
|
| 39 |
+
`air-units`, `power-signals` (today inert), `scheduled-event`,
|
| 40 |
+
`dynamic-deadline`, `RL-adversary` (P3-1).
|
| 41 |
+
|
| 42 |
+
## Engineering shopping list (PRs that unblock the most seeds)
|
| 43 |
+
|
| 44 |
+
Order from cheapest to most expensive; numbers in parentheses are how
|
| 45 |
+
many seeds each PR unblocks.
|
| 46 |
+
|
| 47 |
+
1. **`forbidden_tools` + `tool_violation` signal** (P2-1) โ unblocks
|
| 48 |
+
~6 seeds (Group I + B3).
|
| 49 |
+
2. **`raider` bot** โ unblocks ~4 (B1 + Group F defenses + C2).
|
| 50 |
+
3. **`then:[A,B]` happened-before composite** (P2-3) โ unblocks ~5
|
| 51 |
+
(B2 + Group G/I/N).
|
| 52 |
+
4. **`own_building_lost_gte` + `building_destroyed_gte`** (P2-2) โ
|
| 53 |
+
unblocks ~3 (C3 + C2 + C5).
|
| 54 |
+
5. **`region_held_for_ticks`** โ strengthens B4 + several defense
|
| 55 |
+
seeds.
|
| 56 |
+
6. **`pincer` / `feinter` bots** โ unblocks C3, C4, several adv-*.
|
| 57 |
+
7. **Mid-episode scripted-event hook** โ unblocks C5 + 4 robustness
|
| 58 |
+
seeds + surprise-flank.
|
| 59 |
+
8. **Power-signals fix** โ unblocks 7+ seeds in Groups A/M.
|
| 60 |
+
9. **MCV `deploy` order** โ unblocks ALL Group A `mcv-*` (7+ seeds).
|
| 61 |
+
10. **`capture_actor` order** โ unblocks Group K spec-engineer-*.
|
| 62 |
+
11. **Engineer-rebuild order** โ unblocks several robustness seeds.
|
| 63 |
+
12. **`apc` transport completeness** โ unblocks Group L.
|
| 64 |
+
13. **Water mapgen + `lst`** โ unblocks Group L amphibious + island.
|
| 65 |
+
14. **Air units (`yak`/`mig`/`heli`)** โ unblocks anti-air,
|
| 66 |
+
paradrop, air-ground coord.
|
| 67 |
+
15. **Stealth/`spy`/`thief` mechanics** โ unblocks spec-spy / spec-thief.
|
| 68 |
+
16. **Superweapons** (nuke / iron-curtain / chronosphere) โ Group K.
|
| 69 |
+
17. **Live RL adversary slot (P3-1)** โ unblocks C1, C4, all
|
| 70 |
+
adv-*-race seeds.
|
| 71 |
+
|
| 72 |
+
---
|
| 73 |
+
|
| 74 |
+
# Backlog seeds organized by family
|
| 75 |
+
|
| 76 |
+
Each seed has: `id` โ cell โ real-world anchor // benchmark anchor โ
|
| 77 |
+
one-line predicate sketch โ engineering tag(s).
|
| 78 |
+
|
| 79 |
+
## Group A โ Base building & MCV (25)
|
| 80 |
+
|
| 81 |
+
Largely engine-gated on `deploy`; pre-`deploy` proxies via pre-placed
|
| 82 |
+
`fact` markers + build-radius creep.
|
| 83 |
+
|
| 84 |
+
- `mcv-deploy-and-build` โ Opening MCVโfact site choice // critical-infra siting; SC2LE BO โ `has_building:fact` AND deploy-tick โค T AND `building_total_gte:3` โ ENG:deploy + YAML
|
| 85 |
+
- `mcv-deploy-defensible-site` โ Opening site under threat // resilience-first siting; MicroRTS terrain-aware โ `building_in_region:{good_terrain,fact,1}` AND survival โ ENG:deploy + YAML
|
| 86 |
+
- `mcv-deploy-near-resource` โ Opening site near ore // econ-first siting; SC2LE econ โ `building_in_region:{near_ore,fact,1}` AND `economy_value_gte:N` โ ENG:deploy + YAML
|
| 87 |
+
- `mcv-deploy-relocate-under-pressure` โ Mid conyard pack + relocate // BCP relocation; PlanBench replanning โ `building_in_region:{new_site,fact,1}` after `after_ticks:T_loss` โ ENG:deploy+undeploy + YAML
|
| 88 |
+
- `mcv-deploy-second-base` โ Mid 2nd MCV โ 2nd base // multi-site ramp; MicroRTS expansion โ 2 facts in 2 regions โ ENG:deploy + YAML
|
| 89 |
+
- `mcv-deploy-third-base` โ Mid-Late 3rd MCV โ 3rd base // tri-base macro; SC2 3-base โ 3 facts in 3 regions โ ENG:deploy + YAML
|
| 90 |
+
- `mcv-deploy-evac-and-resite` โ Mid evacuate failing base // disaster recovery; ScienceWorld โ fact razed THEN new fact built โ ENG:deploy + YAML
|
| 91 |
+
- `build-radius-creep-to-objective` โ Mid walk base toward enemy // FOB; SC2 forward pylon โ `building_in_region:{near_enemy,pbox,1}` โ YAML
|
| 92 |
+
- `build-radius-creep-bridge-island` โ Mid-Late creep over bridge // staging expansion; MicroRTS terrain โ `building_in_region:{island,fact,1}` โ ENG:bridge-aware-mapgen + YAML
|
| 93 |
+
- `build-power-online-first` โ Opening `has_building:powr` before all else // grid bring-up sequencing; PlanBench ordering โ `then:[powr, _]` โ PRED:then + YAML
|
| 94 |
+
- `build-defensive-tower-line` โ Mid 4 pbox in line covering choke // perimeter design; ERQA spatial โ `building_count_gte:{type:pbox,n:4}` + `building_in_region:{choke,pbox,4}` โ YAML
|
| 95 |
+
- `build-defensive-tower-cluster` โ Mid pbox cluster at bottleneck // concentrated defense; ERQA โ `building_in_region:{bottleneck,pbox,4}` โ YAML
|
| 96 |
+
- `build-defensive-skirt-corners` โ Mid corner-mounted defenses // distributed defense; MicroRTS โ `building_in_region` 4x corner regions โ YAML
|
| 97 |
+
- `build-spacing-anti-splash` โ Mid space buildings to survive splash // resilience design; robustness โ `min_separation_gte:N` โ PRED:min_separation + YAML
|
| 98 |
+
- `build-anti-air-positioning` โ Mid SAM coverage // anti-air planning; SC2 antiair โ `building_in_region` of `sam` โ ENG:air-units + YAML
|
| 99 |
+
- `build-engineer-rebuild-after-loss` โ Mid engineer rebuilds razed building // resilience drill; PlanBench replanning โ building re-presence after loss โ ENG:engineer-rebuild-order + YAML
|
| 100 |
+
- `build-sequence-tech-fastest` โ Opening BO hits `weap` in min turns // BOM-optimal; PlanBench cost-optimal โ `then:[tent, weap]` AND `within_ticks:T_tight` โ PRED:then + YAML
|
| 101 |
+
- `build-sequence-tech-cheapest` โ Opening BO hits `weap` cheapest // BOM-cost; PlanBench cost-optimal โ cash-bound โ YAML
|
| 102 |
+
- `build-sequence-tech-most-resilient` โ Mid tech via redundant prereqs // resilience; PlanBench robust โ 2x prereq buildings โ PRED:then + YAML
|
| 103 |
+
- `build-power-budget-online` โ Opening keep `power_surplus_gte:0` // grid bring-up; SC2 power โ ENG:power-signals + YAML
|
| 104 |
+
- `build-power-down-defensive` โ Mid `power_down` non-essentials under attack // load-shedding; ops runbook โ YAML
|
| 105 |
+
- `build-repair-priority-under-fire` โ Mid repair structures in correct order // repair triage; DR โ YAML
|
| 106 |
+
- `build-sell-and-rebuild-elsewhere` โ Mid `sell` weakened, rebuild defended // capital reallocation; finance โ YAML
|
| 107 |
+
- `build-rally-point-management` โ Opening set rally forward // production logistics; warehouse SLA โ YAML
|
| 108 |
+
- `build-production-throughput-multibuilding` โ Mid 2x `weap` parallel queue // throughput; queueing theory โ YAML
|
| 109 |
+
|
| 110 |
+
## Group B โ Multi-front base building (12)
|
| 111 |
+
|
| 112 |
+
- `mfb-two-base-simultaneous` โ Mid build base-2 while base-1 builds // multi-site ramp; SC2 sim-base โ 2 facts within tight window โ YAML
|
| 113 |
+
- `mfb-base-1-defend-base-2-build` โ Mid defend live + build forward // ops continuity during expansion; DR siting โ YAML
|
| 114 |
+
- `mfb-third-base-against-clock` โ Mid-Late 3rd base before deadline // greedy expansion; SC2 3-base timing โ YAML
|
| 115 |
+
- `mfb-relocate-conyard-under-pressure` โ Mid pack + move conyard // BCP; PlanBench replanning โ ENG:deploy/undeploy + YAML
|
| 116 |
+
- `mfb-evacuate-base-deploy-elsewhere` โ Mid abandon base, redeploy MCV // disaster recovery; ScienceWorld โ ENG:deploy + YAML
|
| 117 |
+
- `mfb-tech-base-vs-economy-base` โ Mid-Late role-specialized bases // specialization; industrial โ YAML
|
| 118 |
+
- `mfb-island-base-amphibious` โ Late build base across water // amphibious staging; RoboMaster ship-to-shore โ ENG:transport+amphib-mapgen + YAML
|
| 119 |
+
- `mfb-forward-base-near-enemy-corner` โ Mid-Late FOB // SC2 forward production โ YAML
|
| 120 |
+
- `mfb-redundant-tech-buildings` โ Mid 2x `weap` for resilience // redundancy; robust planning โ YAML
|
| 121 |
+
- `mfb-mirror-base-east-west` โ Mid symmetric multi-base // distributed-systems mirror; raft replicas โ YAML
|
| 122 |
+
- `mfb-rotating-production-pressure` โ Mid-Late alternate production under raid // load balancing; queueing โ PRED:then + YAML
|
| 123 |
+
- `mfb-supply-line-link-between-bases` โ Mid-Late connect 2 bases via patrol // logistics route; supply-chain โ YAML
|
| 124 |
+
|
| 125 |
+
## Group C โ Defense positioning & surprise reaction (16)
|
| 126 |
+
|
| 127 |
+
- `def-position-expected-direction` โ Mid fortify N (intel says N) // pre-positioned defense; ERQA spatial commit โ YAML
|
| 128 |
+
- `def-position-revealed-direction` โ Mid fortify after scout reveals // adaptive defense; PlanBench replanning โ YAML
|
| 129 |
+
- `def-surprise-flank-react` โ Mid intel said N, enemy comes S // surprise reaction; adversarial robustness โ ENG:enemy-spawn-misdirection + YAML
|
| 130 |
+
- `def-pre-position-mobile-reserve` โ Mid mobile reserve in center // reserve doctrine; military โ YAML
|
| 131 |
+
- `def-tower-line-vs-cluster` โ Mid choose spread vs concentration // defense topology; graph min-cut โ YAML
|
| 132 |
+
- `def-walls-vs-towers` โ Mid passive vs active mitigation // security architecture โ ENG:walls + YAML
|
| 133 |
+
- `def-engineer-repair-under-fire` โ Mid which tower to repair under fire // triage; DR โ YAML
|
| 134 |
+
- `def-stance-mgmt-hold-then-attack` โ Mid hold-fireโreturn-fire timing // ROE drill โ YAML
|
| 135 |
+
- `def-while-building` โ Mid defend with workforce while building defenses // ops continuity โ YAML
|
| 136 |
+
- `def-multi-direction` โ Mid defend 3 lanes // distributed defense; graph min-cut โ YAML
|
| 137 |
+
- `def-in-depth` โ Mid-Late front + 2nd line vs single thick // defense-in-depth; security doctrine โ YAML
|
| 138 |
+
- `def-with-ambush` โ Mid fog ambush vs static // active deception; SC2 hidden โ YAML
|
| 139 |
+
- `def-evacuation` โ Mid-Late evacuate overwhelmed position // BCP; emergency mgmt โ YAML
|
| 140 |
+
- `def-retreat-and-rebuild` โ Mid-Late concede + rebuild deeper // strategic withdrawal; CICERO retreat โ YAML
|
| 141 |
+
- `def-anti-air-positioning` โ Late SAM placement // anti-air; SC2 โ ENG:air-units + YAML
|
| 142 |
+
- `def-tower-rush-counter` โ Opening enemy tower-rushes you // tower-rush defense; SC2 cannon-rush counter โ BOT:tower-rusher + YAML
|
| 143 |
+
|
| 144 |
+
## Group D โ Combat micro & tactics (20)
|
| 145 |
+
|
| 146 |
+
- `combat-focus-fire-priority` โ Mid which enemy to kill first // target prioritization; SC2 focus-fire โ YAML
|
| 147 |
+
- `combat-kite-jeep-vs-tank` โ Mid kite ranged // micro; SC2 kiting โ YAML
|
| 148 |
+
- `combat-stance-mgmt-attack` โ Mid switch to attack-anything // ROE; SC2 stance โ YAML
|
| 149 |
+
- `combat-formation-tank-wedge` โ Mid tank wedge // formation; military โ YAML
|
| 150 |
+
- `combat-formation-infantry-screen-tanks` โ Mid infantry screens armor // combined-arms; military โ YAML
|
| 151 |
+
- `combat-retreat-after-engagement` โ Mid disengage timing // tactical withdrawal; SC2 โ YAML
|
| 152 |
+
- `combat-flanking-attack` โ Mid flank vs head-on // flank maneuver; military โ YAML
|
| 153 |
+
- `combat-pincer-coordination` โ Mid two-prong // pincer; SC2 multi-prong โ YAML
|
| 154 |
+
- `combat-bait-counter-attack` โ Mid lure + flank-counter // bait; SC2 bait โ BOT:guard + YAML (idiom proven)
|
| 155 |
+
- `combat-divide-and-conquer` โ Mid split enemy via feint // CICERO โ YAML
|
| 156 |
+
- `combat-protect-vip-escort` โ Mid escort VIP // diplomatic escort โ PRED:`unit_type_count_gte:{vip,1}`-survival + YAML
|
| 157 |
+
- `combat-suicide-charge-mission` โ Mid sacrifice everything for X // forlorn hope; military โ YAML
|
| 158 |
+
- `combat-prevent-retreat` โ Mid cut off enemy retreat // encirclement; military โ YAML
|
| 159 |
+
- `combat-defensive-perimeter-around-vip` โ Mid perimeter // VIP cordon โ YAML
|
| 160 |
+
- `combat-skirmish-then-disengage` โ Mid bleed and pull back // skirmisher; SC2 โ YAML
|
| 161 |
+
- `combat-vehicle-vs-infantry-counter` โ Mid tank vs inf // RPS counter; SC2 hard counter โ YAML
|
| 162 |
+
- `combat-rocket-soldier-anti-vehicle` โ Mid e3 vs tank // RPS; SC2 โ YAML
|
| 163 |
+
- `combat-tank-vs-tank-engagement` โ Mid mirror combat // SC2 mirror micro โ YAML
|
| 164 |
+
- `combat-amphibious-landing` โ Mid-Late APC across water // amphibious assault; D-Day โ ENG:transport+water-mapgen + YAML
|
| 165 |
+
- `combat-attack-from-behind-fog` โ Mid emerge from fog // surprise attack; SC2 hidden โ YAML
|
| 166 |
+
|
| 167 |
+
## Group E โ Scouting / Perception (15)
|
| 168 |
+
|
| 169 |
+
- `scout-far-frontier` โ Early long-distance scout // recon; ERQA โ YAML
|
| 170 |
+
- `scout-and-survive` โ Early scout returns alive // scout-preserve; SC2 worker scout โ YAML
|
| 171 |
+
- `scout-multiple-fog-areas` โ Early parallel scout // Watch-And-Help โ YAML
|
| 172 |
+
- `scout-detect-enemy-tech` โ Early-Mid identify enemy via buildings // tech-read; PlanBench obs โ YAML
|
| 173 |
+
- `scout-detect-base-direction` โ Early which corner has enemy // direction-read; ERQA โ YAML
|
| 174 |
+
- `scout-count-defenders` โ Early-Mid exact defender count // census; POMDP โ YAML
|
| 175 |
+
- `scout-discover-hidden-base` โ Mid off-axis hidden base // deep recon; CICERO โ YAML
|
| 176 |
+
- `scout-detect-incoming-army` โ Mid early-warn // intrusion detection โ YAML
|
| 177 |
+
- `scout-detect-air-threat` โ Late air detection // anti-air recon โ ENG:air-units + YAML
|
| 178 |
+
- `scout-detect-stealth-units` โ Late stealth detection // SC2 detector โ ENG:stealth-units + YAML
|
| 179 |
+
- `scout-map-reveal-percent-target` โ Early reveal N% // coverage; ERQA โ YAML
|
| 180 |
+
- `scout-and-report` โ Early-Mid reveal then return // recon-report cycle; mil intel โ YAML
|
| 181 |
+
- `scout-jeep-vs-infantry-cost-effective` โ Early cheap fast scout // econ scout; SC2 worker-vs-Reaper โ YAML
|
| 182 |
+
- `scout-cycle-keep-info-fresh` โ Mid continuous cadence // info staleness; intel ops โ PRED:then-repeated + YAML
|
| 183 |
+
- `scout-info-blackout-react` โ Mid fog blanket then enemy revealed // dynamic perception; adversarial โ ENG:fog-shift-hook + YAML
|
| 184 |
+
|
| 185 |
+
## Group F โ Economy / Resources (16)
|
| 186 |
+
|
| 187 |
+
Mostly unblocked by S0/S1 (task #14).
|
| 188 |
+
|
| 189 |
+
- `econ-startup-from-scratch` โ Opening first proc + harv // greenfield econ; SC2 first base โ YAML
|
| 190 |
+
- `econ-harvester-pathing-optimization` โ Opening harv route choice // logistics; queueing โ PRED:`econ_rate_gte` + YAML
|
| 191 |
+
- `econ-multi-patch-allocation` โ Mid which ore patches // resource allocation; OR โ YAML
|
| 192 |
+
- `econ-far-patch-vs-near-patch` โ Mid travel vs throughput // econ tradeoff; OR โ YAML
|
| 193 |
+
- `econ-silo-vs-spend` โ Mid store or spend // capital allocation; PlanBench โ YAML
|
| 194 |
+
- `econ-replace-dead-harvester` โ Mid rebuild dead harv // resilience; PlanBench replanning โ YAML
|
| 195 |
+
- `econ-protect-harvester-route` โ Mid defend harv path // econ defense; SC2 harass defense โ YAML
|
| 196 |
+
- `econ-cash-reserve-management` โ Mid never zero cash // financial reserve โ PRED:`cash_gte` + YAML
|
| 197 |
+
- `econ-target-cash-amount-by-deadline` โ Mid cash bar by deadline // budget by date; finance โ YAML
|
| 198 |
+
- `econ-contention-with-enemy` โ Mid shared ore patch // contested resource; game theory โ YAML
|
| 199 |
+
- `econ-recover-from-zero-cash` โ Mid start from 0 cash // bankruptcy comeback โ YAML
|
| 200 |
+
- `econ-quantitative-vs-qualitative-spend` โ Mid many cheap vs few expensive // unit composition; OR โ YAML
|
| 201 |
+
- `econ-burn-rate-management` โ Mid spend at target rate // runway mgmt; startup finance โ PRED:`spend_rate_lte` + YAML
|
| 202 |
+
- `econ-buy-vs-build-decision` โ Mid cash now or save for tech // PlanBench โ YAML
|
| 203 |
+
- `econ-resource-trade-with-self` โ Mid-Late balance ore vs cash // commodity hedging; finance โ YAML
|
| 204 |
+
- `econ-overflow-to-silos` โ Mid silo-required throughput // capacity planning; backpressure โ YAML
|
| 205 |
+
|
| 206 |
+
## Group G โ Long-horizon / Multi-phase (12)
|
| 207 |
+
|
| 208 |
+
- `lh-opening-to-tech-to-army` โ Late 4-phase chain // long-horizon; SC2 macro โ YAML
|
| 209 |
+
- `lh-opening-to-defense-to-counter` โ Late defense โ counter // counter-attack timing; SC2 timing โ YAML
|
| 210 |
+
- `lh-tech-pivot-attack` โ Late tech โ pivot โ attack // adaptive macro; PlanBench โ YAML
|
| 211 |
+
- `lh-econ-army-victory` โ Late econ โ army โ victory // SC2 macro โ YAML
|
| 212 |
+
- `lh-defense-tech-second-base` โ Late defend โ tech โ expand // SC2 secure-expand โ YAML
|
| 213 |
+
- `lh-scout-react-counter` โ Late scout โ react โ counter // CICERO โ YAML
|
| 214 |
+
- `lh-100-turn-marathon-survival` โ Very-Late survive 100 turns // lmgame-Bench multi-hour โ YAML (max_turns:100+)
|
| 215 |
+
- `lh-build-army-coordinate-multifront-attack` โ Late assemble + coordinate // operational planning; military โ YAML
|
| 216 |
+
- `lh-progression-stage-locked` โ Late each stage gates next // PERT; PlanBench โ PRED:then + YAML
|
| 217 |
+
- `lh-credit-only-final-phase` โ Late sparse reward // long-horizon RL โ YAML
|
| 218 |
+
- `lh-multi-checkpoint-5-plus` โ Late 5+ checkpoints // sequenced ops; PlanBench โ PRED:waypoint_sequence-extended + YAML
|
| 219 |
+
- `lh-recovery-after-mid-game-loss` โ Late comeback // SC2 comeback โ ENG:scheduled-event + YAML
|
| 220 |
+
|
| 221 |
+
## Group H โ Adversarial / Game-theoretic (12)
|
| 222 |
+
|
| 223 |
+
- `adv-counter-strategy-read` (= C1 from plan)
|
| 224 |
+
- `adv-feint-handling` (= C4 from plan)
|
| 225 |
+
- `adv-bluff-detection` โ Mid commit vs hesitate // poker-bench โ ENG:RL-adversary + YAML
|
| 226 |
+
- `adv-rps-counter-pick` โ Mid unit-counter selection // RPS; game theory โ YAML
|
| 227 |
+
- `adv-tempo-vs-adversary` โ Mid-Late out-tempo // TextStarCraft II โ ENG:RL-adversary + YAML
|
| 228 |
+
- `adv-economy-race` โ Late out-econ // SC2 econ race โ ENG:RL-adversary + YAML
|
| 229 |
+
- `adv-army-race` โ Late out-army // SC2 army race โ ENG:RL-adversary + YAML
|
| 230 |
+
- `adv-tech-race` โ Late out-tech // SC2 tech race โ ENG:RL-adversary + YAML
|
| 231 |
+
- `adv-base-trade-race` (= C3 from plan)
|
| 232 |
+
- `adv-asymmetric-weaker-must-win` โ Mid outnumbered must-win // underdog; RTS asymmetric โ YAML
|
| 233 |
+
- `adv-asymmetric-stronger-no-loss` โ Mid dominate without loss // overdog discipline; CICERO โ YAML
|
| 234 |
+
- `adv-mirror-match-best-strategy` โ Late mirror best-of // SC2 mirror โ ENG:RL-adversary + YAML
|
| 235 |
+
|
| 236 |
+
## Group I โ Procedural / Tool-use compliance (10)
|
| 237 |
+
|
| 238 |
+
- `proc-strict-toolban-fidelity` โ Cross BFCL relevance // BFCL V4 โ PRED:forbidden_tools + YAML
|
| 239 |
+
- `proc-strict-toolban-under-pressure` (= B3 from plan)
|
| 240 |
+
- `proc-checklist-no-deviation` โ Cross exact ordered checklist // SOP compliance; IFBench โ PRED:waypoint_sequence strict + YAML
|
| 241 |
+
- `proc-ordered-action-strict` โ Cross exact action sequence // PlanBench strict ordering โ PRED:then + YAML
|
| 242 |
+
- `proc-no-attack-passive-only` โ Cross only move+stop // ROE; mil โ PRED:forbidden_tools + YAML
|
| 243 |
+
- `proc-only-build-no-combat` โ Cross only build/place // role-based access โ PRED:forbidden_tools + YAML
|
| 244 |
+
- `proc-only-defend-no-attack` โ Cross only defensive moves // ROE; mil โ PRED:forbidden_tools + YAML
|
| 245 |
+
- `proc-tool-use-with-distractor` โ Cross ignore distracting tool // ฯยฒ-bench โ YAML
|
| 246 |
+
- `proc-tool-use-multi-distractor` โ Cross ignore multiple distractors // ฯยฒ-bench โ YAML
|
| 247 |
+
- `proc-instruction-following-edge-case` โ Cross instruction-edge // IFBench โ YAML
|
| 248 |
+
|
| 249 |
+
## Group J โ Robustness / Edge cases (8)
|
| 250 |
+
|
| 251 |
+
- `rob-unit-loss-recovery` โ Mid recover from N losses // resilience; PlanBench replanning โ YAML
|
| 252 |
+
- `rob-building-loss-rebuild` (= C5 from plan)
|
| 253 |
+
- `rob-cash-depletion-recovery` โ Mid recover from 0 cash // bankruptcy โ YAML
|
| 254 |
+
- `rob-power-failure-recovery` โ Mid recover from brownout // grid restoration; DR โ ENG:power-signals + YAML
|
| 255 |
+
- `rob-multiple-simultaneous-pressures` โ Mid econ + def + tech under pressure // overwhelmed ops; triage โ YAML
|
| 256 |
+
- `rob-unexpected-enemy-spawn` โ Mid spawn mid-episode // surprise; adversarial robustness โ ENG:scheduled-spawn + YAML
|
| 257 |
+
- `rob-objective-change-midway` โ Mid win-condition shifts at T // goal-conditional RL โ ENG:objective-swap-hook + YAML
|
| 258 |
+
- `rob-deadline-shortened-midway` โ Mid deadline brought forward // schedule compression; project mgmt โ ENG:dynamic-deadline-hook + YAML
|
| 259 |
+
|
| 260 |
+
## Group K โ Special abilities (engine-gated, 10)
|
| 261 |
+
|
| 262 |
+
- `spec-engineer-capture` โ Mid capture enemy `proc` // industrial espionage; SC2 spy steal โ ENG:capture_actor + YAML
|
| 263 |
+
- `spec-engineer-mass-capture` โ Mid capture all of enemy tech // takeover; CICERO โ ENG:capture_actor + YAML
|
| 264 |
+
- `spec-spy-infiltrate` โ Mid reveal enemy tech via spy // intel infiltration; SC2 scan โ ENG:spy + YAML
|
| 265 |
+
- `spec-thief-steal-resources` โ Mid steal enemy cash // econ sabotage; SC2 worker raid โ ENG:thief + YAML
|
| 266 |
+
- `spec-medic-heal-coordination` โ Mid medic heals infantry // SC2 medivac โ ENG:medic + YAML
|
| 267 |
+
- `spec-airborne-paradrop` โ Late drop reinforcements // SC2 paradrop โ ENG:airborne + YAML
|
| 268 |
+
- `spec-nuke-strike` โ Very-late superweapon // SC2 nuke โ ENG:superweapon + YAML
|
| 269 |
+
- `spec-iron-curtain-buff` โ Late invuln buff // SC2 cloak โ ENG:iron-curtain + YAML
|
| 270 |
+
- `spec-chronosphere-teleport` โ Late unit teleport // SC2 warpgate โ ENG:chronosphere + YAML
|
| 271 |
+
- `spec-spy-detect-tech` โ Mid spy + tech reveal // counterintel; CICERO โ ENG:spy + YAML
|
| 272 |
+
|
| 273 |
+
## Group L โ Transport / Amphibious (6)
|
| 274 |
+
|
| 275 |
+
- `transport-apc-deliver-infantry` โ Mid APC delivers infantry // last-mile delivery; logistics โ ENG:apc-transport + YAML
|
| 276 |
+
- `transport-amphibious-landing` โ Mid-Late LST water cross // amphibious; D-Day โ ENG:water-mapgen+lst + YAML
|
| 277 |
+
- `transport-evacuate-by-transport` โ Mid evac units // BCP evacuation; emergency โ ENG:apc-transport + YAML
|
| 278 |
+
- `transport-multiple-trips` โ Mid ferry multiple loads // queueing โ ENG:apc-transport + YAML
|
| 279 |
+
- `transport-protect-en-route` โ Mid escort transport // convoy escort; mil โ ENG:apc-transport + YAML
|
| 280 |
+
- `transport-amphibious-island-base` โ Mid-Late build on island // amphibious base; strategic geography โ ENG:water-mapgen+lst + YAML
|
| 281 |
+
|
| 282 |
+
## Group M โ Power / Maintenance (7)
|
| 283 |
+
|
| 284 |
+
Most are gated on the engine power-signals fix (`power_surplus_gte`
|
| 285 |
+
is currently inert โ see CLAUDE.md footguns).
|
| 286 |
+
|
| 287 |
+
- `power-budget-online` โ Opening keep `power_surplus_gte:0` // grid bring-up; SC2 power โ ENG:power-signals + YAML
|
| 288 |
+
- `power-down-non-essential-under-attack` โ Mid load-shed // ops runbook โ ENG:power-signals + YAML
|
| 289 |
+
- `power-restoration-order-after-loss` โ Mid restore in correct order // utility restoration; DR โ ENG:power-signals + YAML
|
| 290 |
+
- `power-tech-vs-defense-conflict` โ Mid limited power: tech vs def // capacity allocation; OR โ ENG:power-signals + YAML
|
| 291 |
+
- `maint-repair-priority-order` โ Mid repair triage // DR โ YAML
|
| 292 |
+
- `maint-sell-and-recoup-cash` โ Mid sell + recoup // capital reallocation; finance โ YAML
|
| 293 |
+
- `maint-rebuild-from-power-failure` โ Mid restore after brownout // DR; utility โ ENG:power-signals + YAML
|
| 294 |
+
|
| 295 |
+
## Group N โ Coordination & Multi-agent (8)
|
| 296 |
+
|
| 297 |
+
- `coord-squad-handoff` โ Mid squad A to squad B handoff // multi-agent handoff; Watch-And-Help โ YAML
|
| 298 |
+
- `coord-relay-attack` โ Mid A attacks, B follows up // relay; SMAC โ YAML
|
| 299 |
+
- `coord-mutual-support` โ Mid stay in mutual support range // mil doctrine โ PRED:max_inter_unit_distance_lte + YAML
|
| 300 |
+
- `coord-converge-on-target` โ Mid 3 squads converge // SMAC โ YAML
|
| 301 |
+
- `coord-cover-and-move` โ Mid A covers while B moves // fire-and-maneuver; mil โ YAML
|
| 302 |
+
- `coord-diversionary-attack` โ Mid A diverts, B attacks real target // CICERO โ YAML
|
| 303 |
+
- `coord-air-ground-combined` โ Late (air) combined-arms // mil โ ENG:air-units + YAML
|
| 304 |
+
- `coord-naval-ground-combined` โ Late (naval) combined naval-ground // D-Day โ ENG:naval + YAML
|
| 305 |
+
|
| 306 |
+
## Group O โ Time-pressure variants (8)
|
| 307 |
+
|
| 308 |
+
(Each is a "+ tight clock" variant of an existing scenario family.)
|
| 309 |
+
|
| 310 |
+
- `tp-rush-objective-very-fast` โ Mid speedrun objective // speedrun; human speedrun โ YAML
|
| 311 |
+
- `tp-rush-objective-tighter` โ Mid tighter speedrun โ YAML
|
| 312 |
+
- `tp-rush-multi-objective` โ Mid speedrun 2 objectives // speedrun parallel โ YAML
|
| 313 |
+
- `tp-survive-N-turns` โ Mid survive until clock // survival; SC2 survival โ YAML
|
| 314 |
+
- `tp-survive-and-grow` โ Mid survive + grow // ops continuity; SRE โ YAML
|
| 315 |
+
- `tp-survive-and-strike-at-window` โ Mid survive then strike // SC2 timing window โ PRED:then + YAML
|
| 316 |
+
- `tp-decision-under-clock` โ Mid quick correct choice // tense decision; poker โ YAML
|
| 317 |
+
- `tp-pressure-procedural` โ Cross procedural under tight clock // procedure under stress; IFBench โ PRED:forbidden_tools + YAML
|
|
@@ -62,6 +62,19 @@ class ScenarioMeta(BaseModel):
|
|
| 62 |
..., min_length=10, description="Concrete robotics/agentic parallel"
|
| 63 |
)
|
| 64 |
author: str = "unknown"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
# Hygiene (audit): "quarantine" keeps the file on disk and runnable
|
| 66 |
# by explicit --packs, but excludes it from the default eval set so
|
| 67 |
# it doesn't dilute scores / the leaderboard. Used for the
|
|
|
|
| 62 |
..., min_length=10, description="Concrete robotics/agentic parallel"
|
| 63 |
)
|
| 64 |
author: str = "unknown"
|
| 65 |
+
# Real-world / benchmark anchors โ every pack must name at least one
|
| 66 |
+
# external referent (a named benchmark or a real-world capability)
|
| 67 |
+
# so the scenario carries transfer signal, not just RTS novelty.
|
| 68 |
+
# Multi-anchor packs are common (e.g. tool-fidelity โ BFCL V4 +
|
| 69 |
+
# ฯยฒ-bench + IFBench), so this is a list. The list-non-empty rule
|
| 70 |
+
# is suite-enforced by tests/test_benchmark_anchor_required.py.
|
| 71 |
+
benchmark_anchor: list[str] = Field(
|
| 72 |
+
default_factory=list,
|
| 73 |
+
description="Named benchmarks and/or real-world capabilities this "
|
| 74 |
+
"scenario transfers to (e.g. ['MicroRTS rush-defense', "
|
| 75 |
+
"'SC2LE defend-the-cheese tempo', 'incident-response: defend "
|
| 76 |
+
"production infra under live attack']).",
|
| 77 |
+
)
|
| 78 |
# Hygiene (audit): "quarantine" keeps the file on disk and runnable
|
| 79 |
# by explicit --packs, but excludes it from the default eval set so
|
| 80 |
# it doesn't dilute scores / the leaderboard. Used for the
|