yxc20098 commited on
Commit
26e71da
·
1 Parent(s): d2c2f6b

feat(scenario): rob-unexpected-enemy-spawn — surprise 2nd-wave handling (adversarial robustness / military follow-on anchor)

Browse files

Wave-6 ROBUSTNESS / adversarial robustness pack: an obvious 4-unit
hunter cluster (Wave 1) closes in from the centre on the agent's
starting vision row; a SECOND wave was PRE-PLACED at a fog corner
(NE on easy/medium, both NE+SE on hard) and closes in by ~tick 1500
in a hunt pattern from a different bearing than Wave 1. Winning
requires defeating BOTH waves (units_killed_gte 7/8/9), keeping the
fact, AND keeping enough force alive (own_units_gte:2) — so a
forward-rush that chases the visible enemies all the way to the fog
corner (leaving Wave 1 to walk past the unguarded base / screen
attrited at long range) LOSES, and an engage-but-don't-overcommit
disposition WINS.

Pre-placed agent force: 3× 2tnk + 2× e3 (5 defenders total) + fact
+ proc; starting_cash 0 and no tent so the model CANNOT train fresh
infantry — the binding decision is RESERVING the starting force.
Pure-e1 defenders cannot survive the opening hunt-bot rush (verified
empirically — melee'd by 4 e1 hunters in ~200 ticks), so the medium-
tank durability + rocket-soldier range mix is required for the
screen to hold the line.

Discrimination (deterministic scripted, no model / no network):
* stall (observe-only) → LOSS on every level + seed (k=0,
all 5 defenders bled out under attrition before tick 5400)
* forward-rush (attack-move to (90,8) every turn) → LOSS on every
level + seed (chase the corner, base unguarded, screen wiped)
* intended engage-no-overcommit (chase closest visible threat
capped at ≤12 cells from fact) → WIN on every level + seed.

Hard tier: 2 agent spawn_point groups (NORTH base y=12 / SOUTH base
y=28) round-robined by seed. Both Wave-2 fog-corner squads (NE y=8 /
SE y=32) always place since enemy actors ignore spawn_point per
CLAUDE.md; the same-latitude surprise hits FIRST and the opposite-
latitude one is the late follow-on, so a single memorised "expect
Wave 2 from NE" disposition cannot generalise.

Benchmark anchors: adversarial robustness (surprise wave handling),
military follow-on wave doctrine, ops incident-after-incident
triage, SC2 second-attack timing.

Registered in tests/test_hard_tier.py::UPGRADED.
17 pack-specific tests + 2 hard-tier curation tests all pass.

openra_bench/scenarios/packs/rob-unexpected-enemy-spawn.yaml ADDED
@@ -0,0 +1,354 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # rob-unexpected-enemy-spawn.yaml
2
+ #
3
+ # REASONING capability — Wave-6 ROBUSTNESS / adversarial robustness:
4
+ # surprise SECOND wave from an UNEXPECTED direction. The agent has
5
+ # an obvious 4-unit hunter cluster (Wave 1) closing in from the centre
6
+ # on its starting vision row; a SECOND wave was PRE-PLACED at a fog
7
+ # corner (NE on easy/medium, both NE+SE on hard) and closes in by
8
+ # ~tick 1500 in a hunt pattern from a different bearing than Wave 1.
9
+ # Winning requires defeating BOTH waves (`units_killed_gte`),
10
+ # keeping the base (`has_building:fact`), AND keeping enough force
11
+ # alive (`own_units_gte:2`). The relevant capability is RESERVING
12
+ # capacity for a follow-on attack — keeping the defender screen
13
+ # centred on the base instead of chasing routers to the corner and
14
+ # leaving the base exposed.
15
+ #
16
+ # Real-world / benchmark anchors:
17
+ # - Adversarial robustness: surprise wave / follow-on attack handling.
18
+ # - Military follow-on wave doctrine ("the battle isn't over when
19
+ # the enemy you see is dead"; second-echelon engagement).
20
+ # - SC2 second-attack timing (the canonical "second push" the
21
+ # defender must reserve forces against).
22
+ # - Operations: incident-after-incident triage (the same operator is
23
+ # immediately re-paged on a fresh incident from a different
24
+ # subsystem — the relevant skill is reserving capacity).
25
+ #
26
+ # DISCRIMINATIONS (no defect, no cheat — every brute/lazy play LOSES):
27
+ #
28
+ # • stall (observe only): the 5 pre-placed defenders are stance:3
29
+ # (AttackAnything) and auto-engage hostiles in range, but with no
30
+ # explicit pursuit they sit out of weapon arc as the hunters
31
+ # converge from the centre. Across ~5000 ticks the static garrison
32
+ # bleeds out under attrition; the `units_killed_gte` bar is never
33
+ # met (kills ≈ 0) AND own_units drops to zero → fail-clause fires
34
+ # (LOSS, not a draw, before tick 5400).
35
+ #
36
+ # • forward-rush (chase the visible enemies all the way to the fog
37
+ # corner): the defender screen attack-moves to the Wave-2 fog
38
+ # corner (90, 8); en route Wave 1 from the centre walks PAST the
39
+ # base (5 e1 hunters bee-line to the agent centroid which is now
40
+ # east of the fact); the corner-bound defenders take attrition
41
+ # bombardment from the converged hunter swarm, and either die out
42
+ # (`own_units_gte:1` fail) or fail to reach the kill bar inside
43
+ # the `after_ticks:5401` deadline → LOSS.
44
+ #
45
+ # • intended prepare-for-2nd-wave: engage closest threat with
46
+ # `attack_move`, but CAP the chase distance so defenders never
47
+ # stray more than ~12 cells from the fact. When the 2nd wave
48
+ # closes in from the fog corner the screen is already pulled
49
+ # back to base-adjacent range and intercepts it before fact
50
+ # contact. Both waves dead (kill bar met), fact alive,
51
+ # own_units_gte:2 satisfied → WIN.
52
+ #
53
+ # ENGINE FACTS (CLAUDE.md / SCENARIO_QUALITY.md):
54
+ # - Bot `hunt` (openra-sim/src/scripted_bot.rs::Hunt): each enemy
55
+ # unit pursues the NEAREST agent target. Hunt-driven e1 cover
56
+ # ~1 cell per 1.8 ticks, so the centre cluster at x=80 reaches
57
+ # the defender outpost at x=14 in ~120 ticks (well before tick
58
+ # 200); the fog-corner Wave-2 squads at (90, 8) reach the base
59
+ # latitude in ~tick 400..1000 depending on path congestion —
60
+ # close to the "~tick 1500" spec target.
61
+ # - The defender screen is 3× 2tnk (medium tank, durable + dual-
62
+ # purpose anti-armor / anti-infantry cannon) + 2× e3 (rocket
63
+ # soldier, range 7, high anti-vehicle DPS). 5× e1 alone get
64
+ # overwhelmed by hunt-bot e1s at melee in the first 200 ticks
65
+ # (verified — see git history); the 2tnk durability is needed
66
+ # to absorb the opening rush, the e3 range is needed to bleed
67
+ # the trailing waves.
68
+ # - Pre-placed defenders are stance:3 (AttackAnything) so the
69
+ # "stall" baseline is genuinely passive (no agent input
70
+ # required for any auto-fire) and the FAILURE MODE on stall is
71
+ # a slow attrition bleed across the whole ~5400-tick budget,
72
+ # not an instant first-turn force-wipe.
73
+ # - Inert enemy `fact` marker far east (anti-DRAW): without a
74
+ # persistent enemy actor the engine auto-`done`s on enemy-elim
75
+ # once both waves die, before the win/fail evaluator runs
76
+ # (CLAUDE.md rule 5).
77
+ # - Tick budget: max_turns 90 with `enemy_unit_spotted` +
78
+ # `own_unit_destroyed` interrupts → in practice ~60..75 ticks
79
+ # per re-decision → reachable tick at max_turns ≈ 5400..6750.
80
+ # `after_ticks: 5401` fires inside this band so a non-winner is
81
+ # a real reachable timeout LOSS, never a draw.
82
+ # - `enemy_unit_spotted` interrupt → re-decide the instant Wave 2
83
+ # becomes visible (faster detection of the surprise axis).
84
+ # - `own_unit_destroyed` interrupt → re-decide when a defender
85
+ # falls.
86
+ # - The pre-placed 5 defenders + fact + proc (no tent / cash 0)
87
+ # means the model CANNOT train fresh infantry — the binding
88
+ # decision is RESERVING the starting force, not out-producing
89
+ # the threat.
90
+ # - hard tier: 2 agent spawn_point groups (NORTH base y=12 /
91
+ # SOUTH base y=28) so the corner the surprise wave comes FROM
92
+ # flips per seed; symmetric Wave-2 squads at BOTH corners
93
+ # always place (enemy actors ignore spawn_point — CLAUDE.md),
94
+ # so a single memorised "expect Wave 2 from NE" disposition
95
+ # cannot generalise.
96
+
97
+ meta:
98
+ id: rob-unexpected-enemy-spawn
99
+ title: 'Surprise Second Wave — Reserve Forces for the Follow-On'
100
+ capability: reasoning
101
+ real_world_meaning: >
102
+ Adversarial robustness on the temporal axis: the obvious threat
103
+ (a 4-unit cluster near the agent's starting vision) is just the
104
+ OPENING; a SECOND wave was pre-placed at a fog corner and closes
105
+ in from an unexpected bearing by ~tick 1500 in a hunt pattern.
106
+ Winning requires destroying BOTH waves AND keeping the base AND
107
+ keeping enough force alive. The relevant capability is RESERVING
108
+ capacity for a follow-on attack — not committing every defender
109
+ to the visible battle and chasing routers far from the base.
110
+ "Battle finished, push deep" is the canonical military / SRE
111
+ failure mode this trains against.
112
+ robotics_analogue: >
113
+ Incident-after-incident triage: a fresh page arrives from a
114
+ different subsystem the moment the operator resolves the visible
115
+ incident; the operator must keep enough on-call capacity in
116
+ reserve to handle the follow-on instead of fully committing
117
+ every responder to the first symptom. Adversarial robustness:
118
+ treating the first observable threat as ground truth and
119
+ over-fitting the response to it leaves the system exposed when
120
+ the adversary stages a second engagement from a different angle.
121
+ benchmark_anchor:
122
+ - 'adversarial robustness: surprise wave handling'
123
+ - 'military follow-on wave doctrine'
124
+ - 'ops: incident-after-incident triage'
125
+ - 'SC2 second-attack timing'
126
+ author: openra-bench
127
+
128
+ base_map: rush-hour-arena
129
+ # Cash 0: the model CANNOT train fresh defenders (no tent and no
130
+ # cash) — the binding decision is RESERVING the starting 5-unit
131
+ # garrison for the surprise 2nd wave, not out-producing the threat.
132
+ starting_cash: 0
133
+
134
+ base:
135
+ agent:
136
+ faction: allies
137
+ # Hunt bot — each enemy unit pursues the nearest agent target. The
138
+ # Wave-2 squad starts FAR from any agent unit at a fog corner; once
139
+ # it closes inside vision the agent's defender centroid pulls it in.
140
+ enemy:
141
+ faction: soviet
142
+ bot_type: hunt
143
+ tools:
144
+ - observe
145
+ - move_units
146
+ - attack_unit
147
+ - attack_move
148
+ - stop
149
+ planning: true
150
+ # Re-decide the instant a fresh enemy unit becomes visible (the
151
+ # surprise-wave detection signal) or a defender is lost.
152
+ interrupts:
153
+ enemy_unit_spotted: true
154
+ own_unit_destroyed: true
155
+ termination:
156
+ max_ticks: 12000
157
+ # Default actor list (overridden per-level). fact + proc at the
158
+ # west base, no tent / no cash — the agent cannot train replacements.
159
+ actors:
160
+ - {type: fact, owner: agent, position: [10, 20]}
161
+ - {type: proc, owner: agent, position: [6, 20]}
162
+
163
+ levels:
164
+ # ── EASY ─────────────────────────────────────────────────────────
165
+ # 2nd wave SMALLER (3 e1) than Wave 1 (4 e1). Kills bar 7 — the
166
+ # full kill-able enemy force on this level — so the policy MUST
167
+ # defeat BOTH waves (a forward chase that leaves the centre
168
+ # cluster to bombard the screen at long range cannot reach kills=7
169
+ # before the screen is wiped). max_turns 90 with interrupts
170
+ # → reachable tick ~5400..6750; `after_ticks: 5401` bites well
171
+ # inside this band so non-win is a reachable LOSS.
172
+ easy:
173
+ description: >
174
+ Adversarial robustness — surprise second wave (easy). An obvious
175
+ 4-unit rifle cluster sits at the centre of the arena and closes
176
+ in on your base; a SECOND smaller wave (3 rifles) is pre-placed
177
+ at a fog corner and will close in from an unexpected bearing in
178
+ a hunt pattern by ~tick 1500. Your 5 pre-placed defenders
179
+ (3 medium tanks + 2 rocket soldiers) are your ENTIRE force —
180
+ you have no barracks and no cash so you cannot train
181
+ replacements. To WIN you must, before tick 5400, kill ≥7 enemy
182
+ units across BOTH waves AND still have your construction yard
183
+ (fact) standing AND still have ≥2 of your own units alive.
184
+ Stalling (passive defence — the garrison bleeds out under
185
+ attrition), chasing the visible cluster all the way to the
186
+ fog corner (Wave 1 walks past your base while your screen is
187
+ far afield), or busting the attrition floor (≤3 own units lost)
188
+ all LOSE.
189
+ overrides:
190
+ actors:
191
+ - {type: fact, owner: agent, position: [10, 20]}
192
+ - {type: proc, owner: agent, position: [6, 20]}
193
+ # 5 pre-placed defenders (3× 2tnk + 2× e3), stance:3
194
+ # (AttackAnything — they auto-engage hostiles in weapon arc
195
+ # but do not actively pursue beyond their range). 5× e1 alone
196
+ # cannot survive the opening hunt-bot rush (verified —
197
+ # melee'd in ~200 ticks); the 2tnk durability + e3 reach are
198
+ # both needed for the screen to hold the line.
199
+ - {type: 2tnk, owner: agent, position: [14, 20], stance: 3}
200
+ - {type: 2tnk, owner: agent, position: [14, 18], stance: 3}
201
+ - {type: 2tnk, owner: agent, position: [14, 22], stance: 3}
202
+ - {type: e3, owner: agent, position: [16, 19], stance: 3}
203
+ - {type: e3, owner: agent, position: [16, 21], stance: 3}
204
+ # Wave 1 — obvious 4-unit cluster at the centre on the
205
+ # base latitude. Stance:3 + hunt bot drives them straight
206
+ # at the agent centroid; they arrive at the defender
207
+ # outpost by ~tick 150.
208
+ - {type: e1, owner: enemy, position: [80, 20], stance: 3, count: 4}
209
+ # Wave 2 — SMALLER (3 e1) pre-placed at NE fog corner. Hunt
210
+ # bot closes in from y=8 (out of agent's initial vision row);
211
+ # arrives at base latitude ~tick 600..1000.
212
+ - {type: e1, owner: enemy, position: [90, 8], stance: 3, count: 3}
213
+ # Inert enemy `fact` marker far east (anti-DRAW): keeps the
214
+ # episode alive past the last killed hunter so the win/fail
215
+ # evaluator runs.
216
+ - {type: fact, owner: enemy, position: [120, 20]}
217
+ win_condition:
218
+ all_of:
219
+ - units_killed_gte: 7
220
+ - has_building: fact
221
+ - own_units_gte: 2
222
+ - within_ticks: 5400
223
+ # Non-win must be a real LOSS, not a draw. `after_ticks: 5401`
224
+ # is reachable inside max_turns (in practice ~5400..6750 with
225
+ # interrupts); fact-loss and own-units-zero both fail explicitly.
226
+ # NOTE: WIN requires ≥2 alive (the reserve-capacity test); FAIL
227
+ # fires on full wipe (≥1 alive ensures the episode hasn't ended
228
+ # in total elimination).
229
+ fail_condition:
230
+ any_of:
231
+ - after_ticks: 5401
232
+ - not: {has_building: fact}
233
+ - not: {own_units_gte: 1}
234
+ max_turns: 90
235
+
236
+ # ── MEDIUM ───────────────────────────────────────────────────────
237
+ # +1 controlled variable: 2nd wave EQUAL to first (4 e1) → kills bar 8
238
+ # (the canonical spec value). Same clock, same fail.
239
+ medium:
240
+ description: >
241
+ Adversarial robustness — surprise second wave (medium). An
242
+ obvious 4-unit rifle cluster sits at the centre and closes in
243
+ on your base; a SECOND wave OF EQUAL SIZE (4 rifles) is
244
+ pre-placed at a fog corner and closes in from an unexpected
245
+ bearing in a hunt pattern by ~tick 1500. Your 5 pre-placed
246
+ defenders (3 medium tanks + 2 rocket soldiers) are your ENTIRE
247
+ force — no barracks, no cash. To WIN you must, before tick
248
+ 5400, kill ≥8 enemy units across BOTH waves AND still have
249
+ your construction yard (fact) standing AND still have ≥2 of
250
+ your own units alive. Stalling, chasing the visible cluster
251
+ all the way to the fog corner, or attrition-busting all LOSE.
252
+ overrides:
253
+ actors:
254
+ - {type: fact, owner: agent, position: [10, 20]}
255
+ - {type: proc, owner: agent, position: [6, 20]}
256
+ - {type: 2tnk, owner: agent, position: [14, 20], stance: 3}
257
+ - {type: 2tnk, owner: agent, position: [14, 18], stance: 3}
258
+ - {type: 2tnk, owner: agent, position: [14, 22], stance: 3}
259
+ - {type: e3, owner: agent, position: [16, 19], stance: 3}
260
+ - {type: e3, owner: agent, position: [16, 21], stance: 3}
261
+ # Wave 1 — 4-unit cluster (canonical spec count).
262
+ - {type: e1, owner: enemy, position: [80, 20], stance: 3, count: 4}
263
+ # Wave 2 — EQUAL size (4 e1) at NE fog corner.
264
+ - {type: e1, owner: enemy, position: [90, 8], stance: 3, count: 4}
265
+ - {type: fact, owner: enemy, position: [120, 20]}
266
+ win_condition:
267
+ all_of:
268
+ - units_killed_gte: 8
269
+ - has_building: fact
270
+ - own_units_gte: 2
271
+ - within_ticks: 5400
272
+ fail_condition:
273
+ any_of:
274
+ - after_ticks: 5401
275
+ - not: {has_building: fact}
276
+ - not: {own_units_gte: 1}
277
+ max_turns: 90
278
+
279
+ # ── HARD ─────────────────────────────────────────────────────────
280
+ # +1 controlled variable: 2nd wave from TWO fog corners (NE + SE)
281
+ # so the surprise direction flips per seed via 2 agent spawn_point
282
+ # groups (NORTH base y=12 / SOUTH base y=28). Enemy actors ignore
283
+ # spawn_point (CLAUDE.md) — both Wave-2 squads always place. The
284
+ # AGENT spawn round-robins between NORTH and SOUTH, so the SAME-
285
+ # latitude surprise hits FIRST (closer geometry) and the OPPOSITE-
286
+ # latitude one is the late follow-on. The model cannot memorise a
287
+ # single fog-corner heading.
288
+ #
289
+ # Wave-2 squads are 4 e1 each (4 + 4 = 8 wave-2 units + 4 wave-1 =
290
+ # 12 total) and the kill bar is raised to 9 — both waves together
291
+ # produce more than enough kills if the defender screen survives,
292
+ # but a forward chase or attrition bust cannot reach 9 before the
293
+ # screen is wiped. Same clock; same tick-aligned fail.
294
+ hard:
295
+ description: >
296
+ Adversarial robustness — surprise second wave (hard). The
297
+ obvious 4-unit cluster sits at the centre; TWO Wave-2 squads
298
+ are pre-placed at OPPOSITE fog corners (NE and SE) and the
299
+ one at the SAME latitude as your base arrives FIRST while
300
+ the OPPOSITE corner one is the long-haul late follow-on.
301
+ Your base round-robins between a NORTH and a SOUTH latitude
302
+ so which corner the surprise wave arrives from cannot be
303
+ memorised. Your 5 pre-placed defenders (3 medium tanks + 2
304
+ rocket soldiers) are your ENTIRE force — no barracks, no
305
+ cash. To WIN you must, before tick 5400, kill ≥9 enemy units
306
+ across BOTH waves AND still have your construction yard
307
+ (fact) standing AND still have ≥2 of your own units alive.
308
+ Stalling, chasing one fog corner deep, or attrition bust all
309
+ LOSE.
310
+ overrides:
311
+ actors:
312
+ # ── spawn_point 0 — NORTH base (y=12). ─────────────────────
313
+ - {type: fact, owner: agent, position: [10, 12], spawn_point: 0}
314
+ - {type: proc, owner: agent, position: [6, 12], spawn_point: 0}
315
+ - {type: 2tnk, owner: agent, position: [14, 12], stance: 3, spawn_point: 0}
316
+ - {type: 2tnk, owner: agent, position: [14, 10], stance: 3, spawn_point: 0}
317
+ - {type: 2tnk, owner: agent, position: [14, 14], stance: 3, spawn_point: 0}
318
+ - {type: e3, owner: agent, position: [16, 11], stance: 3, spawn_point: 0}
319
+ - {type: e3, owner: agent, position: [16, 13], stance: 3, spawn_point: 0}
320
+ # ── spawn_point 1 — SOUTH base (y=28). ─────────────────────
321
+ - {type: fact, owner: agent, position: [10, 28], spawn_point: 1}
322
+ - {type: proc, owner: agent, position: [6, 28], spawn_point: 1}
323
+ - {type: 2tnk, owner: agent, position: [14, 28], stance: 3, spawn_point: 1}
324
+ - {type: 2tnk, owner: agent, position: [14, 26], stance: 3, spawn_point: 1}
325
+ - {type: 2tnk, owner: agent, position: [14, 30], stance: 3, spawn_point: 1}
326
+ - {type: e3, owner: agent, position: [16, 27], stance: 3, spawn_point: 1}
327
+ - {type: e3, owner: agent, position: [16, 29], stance: 3, spawn_point: 1}
328
+ # Wave 1 — obvious 4-unit cluster at the centre on the mid
329
+ # latitude (y=20). The hunt bot drives them toward the
330
+ # nearest agent target so the cluster naturally engages
331
+ # whichever base is occupied, after a ~one-second detour
332
+ # toward the mid-latitude.
333
+ - {type: e1, owner: enemy, position: [80, 20], stance: 3, count: 4}
334
+ # Wave 2 — TWO 4-unit squads pre-placed at BOTH fog corners
335
+ # (NE y=8 and SE y=32). Enemy actors ignore spawn_point so
336
+ # BOTH always place; whichever corner is at the same
337
+ # latitude as the agent reaches the base FIRST (the EARLY
338
+ # surprise), the OPPOSITE-latitude squad arrives LATER
339
+ # (the LATE follow-on).
340
+ - {type: e1, owner: enemy, position: [90, 8], stance: 3, count: 4}
341
+ - {type: e1, owner: enemy, position: [90, 32], stance: 3, count: 4}
342
+ - {type: fact, owner: enemy, position: [120, 20]}
343
+ win_condition:
344
+ all_of:
345
+ - units_killed_gte: 9
346
+ - has_building: fact
347
+ - own_units_gte: 2
348
+ - within_ticks: 5400
349
+ fail_condition:
350
+ any_of:
351
+ - after_ticks: 5401
352
+ - not: {has_building: fact}
353
+ - not: {own_units_gte: 1}
354
+ max_turns: 90
tests/test_hard_tier.py CHANGED
@@ -512,6 +512,16 @@ UPGRADED = [
512
  # vs SW→SE on the south edge) flips per seed and a memorised
513
  # single-corner reading order cannot generalise across spawns.
514
  "scout-multiple-fog-areas",
 
 
 
 
 
 
 
 
 
 
515
  ]
516
 
517
  # Consciously NOT spawn-varied, with the reason (keeps the curation
 
512
  # vs SW→SE on the south edge) flips per seed and a memorised
513
  # single-corner reading order cannot generalise across spawns.
514
  "scout-multiple-fog-areas",
515
+ # Wave-6 ROBUSTNESS / adversarial robustness seed — surprise 2nd-
516
+ # wave handling (military follow-on wave doctrine / SC2 second-
517
+ # attack timing / ops incident-after-incident triage anchor). Hard
518
+ # tier defines two agent spawn_point groups (NORTH base y=12 /
519
+ # SOUTH base y=28) round-robined by seed; symmetric Wave-2 squads
520
+ # at BOTH fog corners (NE y=8 / SE y=32) always place (enemy
521
+ # actors don't honour spawn_point — CLAUDE.md), so which corner
522
+ # the surprise wave arrives FROM flips per seed and a memorised
523
+ # "expect Wave 2 from NE" disposition cannot generalise.
524
+ "rob-unexpected-enemy-spawn",
525
  ]
526
 
527
  # Consciously NOT spawn-varied, with the reason (keeps the curation
tests/test_rob_unexpected_enemy_spawn.py ADDED
@@ -0,0 +1,279 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """rob-unexpected-enemy-spawn scenario family, full loop on Rust.
2
+
3
+ Wave-6 ROBUSTNESS / adversarial robustness. The pack tests handling
4
+ of a SURPRISE 2nd wave that closes in from a fog corner from a
5
+ different bearing than the obvious 4-unit Wave-1 cluster.
6
+
7
+ The win predicate makes BOTH waves load-bearing:
8
+
9
+ * `units_killed_gte:{7|8|9}` ⇒ the threat is actually handled (NOT
10
+ just Wave 1 — the bar requires both); a forward-chase that runs
11
+ out of force before both waves are dead can never satisfy this;
12
+ * `has_building:fact` ⇒ the base survives — chasing Wave 2 deep into
13
+ the fog corner while Wave 1 walks past the unguarded base razes
14
+ the fact;
15
+ * `own_units_gte:2` ⇒ a minimum reserve survived — attrition busts
16
+ (forward-rush, stall) get the screen wiped (≥4 lost from 5 →
17
+ fail);
18
+ * `within_ticks:5400` paired with `after_ticks:5401` ⇒ a non-finisher
19
+ is a real reachable timeout LOSS (90 turns × ≤90 ticks/step
20
+ reaches ≥5400 in interrupt mode), never a draw.
21
+
22
+ These tests prove with deterministic scripted policies (no model,
23
+ no network) that:
24
+
25
+ * the intended engage-but-don't-chase-deep policy WINS every level +
26
+ every hard seed (1..4);
27
+ * stall / forward-rush both LOSE every level + every hard seed (a
28
+ real LOSS, not a draw);
29
+ * the `after_ticks` deadline is reachable inside `max_turns`;
30
+ * the hard tier defines ≥2 spawn_point groups (so the fog corner
31
+ the 2nd wave arrives from varies by seed — single-cell
32
+ memorisation cannot generalise).
33
+ """
34
+
35
+ from __future__ import annotations
36
+
37
+ import math
38
+
39
+ import pytest
40
+
41
+ pytest.importorskip("openra_train", reason="Rust env wheel not installed")
42
+ pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
43
+ from openra_bench.eval_core import run_level
44
+ from openra_bench.scenarios import load_pack
45
+ from openra_bench.scenarios.loader import PACKS_DIR, compile_level
46
+
47
+ PACK = PACKS_DIR / "rob-unexpected-enemy-spawn.yaml"
48
+ LEVELS = ("easy", "medium", "hard")
49
+ SEEDS = (1, 2, 3, 4)
50
+
51
+
52
+ # ── scripted policies ─────────────────────────────────────────────────
53
+
54
+
55
+ def _own_ids(rs):
56
+ return [str(u["id"]) for u in (rs.get("units_summary") or [])]
57
+
58
+
59
+ def _fact_xy(rs):
60
+ fact = next(
61
+ (b for b in (rs.get("own_buildings") or []) if b.get("type") == "fact"),
62
+ None,
63
+ )
64
+ if fact is None:
65
+ return None
66
+ return int(fact["cell_x"]), int(fact["cell_y"])
67
+
68
+
69
+ def _visible_enemy_units(rs):
70
+ """List of (x, y) for every VISIBLE enemy infantry (not the inert
71
+ fact marker far east)."""
72
+ raw = rs.get("_raw") or {}
73
+ ep = raw.get("enemy_positions") if isinstance(raw, dict) else None
74
+ out = []
75
+ if isinstance(ep, list):
76
+ for e in ep:
77
+ if isinstance(e, dict) and e.get("actor_type") in ("e1", "e3"):
78
+ out.append((int(e.get("cell_x", 0)), int(e.get("cell_y", 0))))
79
+ return out
80
+
81
+
82
+ def stall(rs, C):
83
+ """Observe-only baseline. The 5 defenders are stance:3 so they
84
+ auto-fire on hostiles in weapon arc, but without any pursuit they
85
+ bleed out under attrition across the full 5400-tick budget
86
+ (k=0, all 5 lost) → fail-clause fires (LOSS, not a draw)."""
87
+ return [C.observe()]
88
+
89
+
90
+ def forward_rush(rs, C):
91
+ """Defeat-first-only-relax taken to its degenerate limit: chase
92
+ the visible enemies all the way to the Wave-2 fog corner (90, 8).
93
+ Wave 1 from the centre walks PAST the base (the agent centroid
94
+ is now far east); the screen takes attrition bombardment from the
95
+ converged hunter swarm and either gets wiped (`own_units_gte:1`
96
+ fail) or fails to meet the kill bar before the `after_ticks:5401`
97
+ deadline → LOSS on every level + seed."""
98
+ own = _own_ids(rs)
99
+ if not own:
100
+ return [C.observe()]
101
+ return [C.attack_move(own, 90, 8)]
102
+
103
+
104
+ def make_intended():
105
+ """Engage the closest visible threat with `attack_move`, but CAP
106
+ the chase distance so the defender screen never strays more than
107
+ ~12 cells from the fact. When the 2nd wave closes in from the
108
+ fog corner the screen is already pulled back to base-adjacent
109
+ range and intercepts before fact contact. Both waves dead, fact
110
+ alive, own_units_gte:2 satisfied → WIN every level + seed."""
111
+
112
+ def policy(rs, C):
113
+ own = _own_ids(rs)
114
+ if not own:
115
+ return [C.observe()]
116
+ xy = _fact_xy(rs)
117
+ if xy is None:
118
+ return [C.attack_move(own, 30, 20)]
119
+ fx, fy = xy
120
+ threats = _visible_enemy_units(rs)
121
+ if threats:
122
+ threats.sort(key=lambda p: math.hypot(p[0] - fx, p[1] - fy))
123
+ ex, ey = threats[0]
124
+ # Reserve-capacity guardrail: don't chase enemies > 12
125
+ # cells from the fact — fall back to a base-adjacent ring.
126
+ if abs(ex - fx) > 12 or abs(ey - fy) > 12:
127
+ return [C.attack_move(own, fx + 6, ey)]
128
+ return [C.attack_move(own, ex, ey)]
129
+ return [C.attack_move(own, fx + 4, fy)]
130
+
131
+ return policy
132
+
133
+
134
+ # ── scenario-shape invariants ─────────────────────────────────────────
135
+
136
+
137
+ def test_pack_compiles_with_three_levels_and_hunt_bot():
138
+ pack = load_pack(PACK)
139
+ assert pack.meta.id == "rob-unexpected-enemy-spawn"
140
+ assert pack.meta.capability == "reasoning"
141
+ assert set(pack.levels) == {"easy", "medium", "hard"}
142
+ # Required-by-spec benchmark anchors (the four named in meta).
143
+ anchors = pack.meta.benchmark_anchor
144
+ assert any("adversarial robustness" in a for a in anchors), anchors
145
+ assert any("follow-on" in a for a in anchors), anchors
146
+ assert any("SC2" in a for a in anchors), anchors
147
+ # Hunt bot must be wired through to the engine for every level.
148
+ for lvl in LEVELS:
149
+ c = compile_level(pack, lvl)
150
+ assert c.map_supported
151
+ enemy = c.scenario.enemy
152
+ bot = getattr(enemy, "bot_type", None) or getattr(enemy, "bot", None)
153
+ assert str(bot).lower() == "hunt", (lvl, bot)
154
+
155
+
156
+ @pytest.mark.parametrize("level", LEVELS)
157
+ def test_every_level_has_a_reachable_timeout_fail(level):
158
+ """Non-win must be a real LOSS: the `after_ticks` fail must be
159
+ strictly below the tick reachable at max_turns (≤90 ticks/step in
160
+ interrupt mode)."""
161
+ c = compile_level(load_pack(PACK), level)
162
+ assert c.fail_condition is not None
163
+ fc = c.fail_condition.model_dump(exclude_none=True)
164
+ deadline = None
165
+ for clause in fc.get("any_of", []) or []:
166
+ if "after_ticks" in clause:
167
+ deadline = int(clause["after_ticks"])
168
+ assert deadline is not None, f"{level}: no after_ticks fail clause"
169
+ reachable = 93 + 90 * (c.max_turns - 1)
170
+ assert deadline < reachable, (
171
+ f"{level}: deadline {deadline} unreachable within "
172
+ f"{c.max_turns} turns (max tick {reachable}) → draw degeneracy"
173
+ )
174
+
175
+
176
+ def test_hard_has_two_spawn_point_groups():
177
+ """Hard-tier contract: ≥2 distinct seed-driven spawn_point groups
178
+ so the fog corner the surprise wave arrives from varies by seed
179
+ (anti-memorisation)."""
180
+ c = compile_level(load_pack(PACK), "hard")
181
+ groups = {
182
+ a.spawn_point for a in c.scenario.actors
183
+ if a.owner == "agent" and a.spawn_point is not None
184
+ }
185
+ assert groups == {0, 1}, groups
186
+ # In-bounds check (rush-hour-arena playable y ≈ 2..38, x ≈ 2..126):
187
+ for a in c.scenario.actors:
188
+ x, y = a.position
189
+ assert 2 <= x <= 126 and 2 <= y <= 38, (a.type, a.position)
190
+
191
+
192
+ def test_starting_cash_zero_for_all_levels():
193
+ """Reserve-capacity discrimination: no cash + no tent = the model
194
+ CANNOT train fresh defenders, so the binding decision is
195
+ RESERVING the starting force, not out-producing the threat."""
196
+ pack = load_pack(PACK)
197
+ assert pack.starting_cash == 0
198
+ # No barracks pre-placed on any level.
199
+ for lvl in LEVELS:
200
+ c = compile_level(pack, lvl)
201
+ agent_buildings = {
202
+ a.type for a in c.scenario.actors
203
+ if a.owner == "agent" and a.type in ("tent", "barr", "hand")
204
+ }
205
+ assert not agent_buildings, (lvl, agent_buildings)
206
+
207
+
208
+ def test_anti_draw_marker_present_on_every_level():
209
+ """Anti-DRAW: an unarmed enemy `fact` marker must persist past
210
+ the last hunter death so the engine doesn't auto-`done` on
211
+ enemy-elim before the win/fail evaluator runs (CLAUDE.md rule 5)."""
212
+ pack = load_pack(PACK)
213
+ for lvl in LEVELS:
214
+ c = compile_level(pack, lvl)
215
+ enemy_facts = [
216
+ a for a in c.scenario.actors
217
+ if a.owner == "enemy" and a.type == "fact"
218
+ ]
219
+ assert len(enemy_facts) >= 1, lvl
220
+
221
+
222
+ # ── solvency: intended WINS every level + every hard seed ────────────
223
+
224
+
225
+ @pytest.mark.parametrize("level", LEVELS)
226
+ def test_intended_engage_no_overcommit_wins_every_level_and_seed(level):
227
+ c = compile_level(load_pack(PACK), level)
228
+ for seed in SEEDS:
229
+ r = run_level(c, make_intended(), seed=seed)
230
+ assert r.outcome == "win", (
231
+ f"{level} seed{seed}: intended engage-no-overcommit play "
232
+ f"must WIN; got {r.outcome} (tick={r.signals.game_tick}, "
233
+ f"kills={r.signals.units_killed}, "
234
+ f"lost={r.signals.units_lost}, "
235
+ f"buildings={r.signals.own_buildings})"
236
+ )
237
+
238
+
239
+ # ── no-cheat: every lazy / brute policy LOSES (not draws) ────────────
240
+
241
+
242
+ @pytest.mark.parametrize("level", LEVELS)
243
+ @pytest.mark.parametrize(
244
+ "policy_name,policy_factory",
245
+ [
246
+ ("stall", lambda: stall),
247
+ ("forward_rush", lambda: forward_rush),
248
+ ],
249
+ )
250
+ def test_lazy_and_overcommit_policies_lose_every_level_and_seed(
251
+ level, policy_name, policy_factory
252
+ ):
253
+ """Stall (passive — bleeds out under attrition) and forward-rush
254
+ (over-commits to the fog corner — base unguarded / screen wiped)
255
+ must BOTH LOSE on every level + every seed — no draw."""
256
+ c = compile_level(load_pack(PACK), level)
257
+ for seed in SEEDS:
258
+ r = run_level(c, policy_factory(), seed=seed)
259
+ assert r.outcome == "loss", (
260
+ f"{level} seed{seed} {policy_name}: must LOSE (real fail, "
261
+ f"not a draw); got {r.outcome} (tick={r.signals.game_tick}, "
262
+ f"kills={r.signals.units_killed}, "
263
+ f"lost={r.signals.units_lost}, "
264
+ f"buildings={r.signals.own_buildings})"
265
+ )
266
+
267
+
268
+ # ── determinism ───────────────────────────────────────────────────────
269
+
270
+
271
+ def test_intended_run_is_deterministic_on_easy():
272
+ c = compile_level(load_pack(PACK), "easy")
273
+ a = run_level(c, make_intended(), seed=3)
274
+ b = run_level(c, make_intended(), seed=3)
275
+ assert (a.outcome, a.turns, a.signals.units_killed) == (
276
+ b.outcome,
277
+ b.turns,
278
+ b.signals.units_killed,
279
+ ), "same seed must be deterministic"