yxc20098 commited on
Commit
e318216
·
1 Parent(s): b0cb710

feat(scenarios): merge PR #8 from yiyu-tian — 4 new packs (rebased onto Wave 1+2)

Browse files

Cherry-picks the 4 scenario packs from PR #8 onto current main (the PR
was based on commit 1400f05 pre-Wave 1; merging as-is would have
deleted the 17 Wave 1+2 packs and 4 engine PRs).

New packs (each adds a transfer-anchored bench cell):

- perception-count-the-threat-small-k (renamed from -the-threat to
avoid collision with Wave 1 A7). Different stress axis from A7:
K=2/3/4 with psychologically non-obvious hidden positions vs A7's
K=4..7 seeded with 2 spawn groups. Both ship; complementary.
Anchor: ERQA / POMDP / ScienceWorld / ICU triage.
- coordination-ordered-rendezvous. Complements A5 staggered-window:
this one tests visit ORDER via waypoint_sequence (n=2) across 2/3/4
waypoints. Anchor: Watch-And-Help / SMAC / PERT / supply-chain
ordered handoff.
- tempo-strike-window. Complements A6 double-window: this one is the
single-band canonical cat-c11 tempo design (forced lull then kill
quota in follow-up window). Anchor: TextStarCraft II / OSWorld /
PlanBench temporal goals.
- risk-blockade-bypass. New cell (no overlap with reasoning-risk-
route): heavy corridor vs light detour with attrition cap making
brute-force infeasible. Anchor: MicroRTS route-risk / PlanBench
cost-vs-survival / field-robotics convoy routing.

Per-pack additions vs PR #8 source:
- meta.benchmark_anchor: list[str] populated (the new schema
requirement we added in Wave 2 foundation, commit 17f024b — post
the PR's base).
- All 4 packs added to tests/test_hard_tier.py::NOT_APPLICABLE with
documented reasons (none use spawn_point variation on hard; each
tightens one decision axis per tier instead).
- perception-count-the-threat-small-k renamed (lowercase 'k' to
satisfy the slug validator; meta.id, test file, and references
all updated).
- author: yiyu-tian on each (credit preserved).

NOT cherry-picked from PR #8:
- The "importorskip on 18 pre-existing test files" infrastructure
commit (cbf068c) — touches files unrelated to the scenarios; the
local suite passes without it (719 pre-PR, 759 post-merge).

Test suite: 719 → 759 passing (+40 — 4 new packs × ~10 tests each).

Closes #8.

openra_bench/scenarios/packs/coordination-ordered-rendezvous.yaml ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Validate: python -m openra_bench.scenarios.validate \
2
+ # openra_bench/scenarios/packs/coordination-staggered-window.yaml
3
+ # See ../CONTRIBUTING.md for the full win-condition grammar.
4
+ #
5
+ # ACTION focus, ORDERED parallel rendezvous. Two/three squads must
6
+ # deliver >=2 units to a sequence of regions IN ORDER (W1 before W2
7
+ # before W3), within a tight overall deadline. Order is enforced by
8
+ # `waypoint_sequence` (stateful latch: W2 only counts after W1 was
9
+ # satisfied at some prior tick). The deadline is tight enough that
10
+ # a single column cannot tour all waypoints in time — parallel
11
+ # columns are required AND the deposit at W1 must come first.
12
+ #
13
+ # Differs from action-multiunit-coordination (simultaneous delivery
14
+ # to N regions at one deadline) on the ORDER axis: this scenario
15
+ # fails if W2 is reached before W1, even if both end up satisfied.
16
+ # Same axis across easy/medium/hard — only the number of waypoints
17
+ # and the deadline tighten.
18
+
19
+ meta:
20
+ id: coordination-ordered-rendezvous
21
+ title: 'Ordered Rendezvous — Deliver Squads to Waypoints In Sequence'
22
+ capability: action
23
+ real_world_meaning: >
24
+ Parallel multi-team delivery under an ordering constraint:
25
+ waypoint A must be reached before waypoint B before waypoint C,
26
+ each requiring two or more units present, all within one tight
27
+ overall window. Skipping or out-of-order delivery does not
28
+ count. Single-column tours blow the deadline; only concurrent
29
+ dispatch with leg-by-leg ordering succeeds.
30
+ robotics_analogue: >
31
+ Sequenced multi-depot fleet dispatch with a precedence
32
+ constraint: team A must arrive at depot 1 before team B may be
33
+ credited for arriving at depot 2 (e.g. handoff / supply chain),
34
+ and the overall window does not permit a single team to do all
35
+ legs in series.
36
+ benchmark_anchor:
37
+ - "Watch-And-Help concurrent multi-agent with sequenced sub-tasks"
38
+ - "SMAC asymmetric squad sequencing"
39
+ - "PERT/CPM precedence-constrained dispatch"
40
+ - "supply-chain ordered handoff: depot A before depot B"
41
+ author: yiyu-tian
42
+
43
+ base_map: rush-hour-arena
44
+
45
+ base:
46
+ agent: {faction: allies}
47
+ enemy: {faction: soviet}
48
+ tools: [move_units, attack_unit, stop_units]
49
+ planning: true
50
+ termination: {max_ticks: 6000}
51
+ actors:
52
+ - {type: 2tnk, owner: agent, position: [5, 8], count: 3}
53
+ - {type: 1tnk, owner: agent, position: [7, 32], count: 3}
54
+
55
+ levels:
56
+ easy:
57
+ description: >
58
+ Two waypoints in sequence: W1 NE (110, 8) before W2 SE
59
+ (110, 32), each requiring >=2 units, all within 2200 ticks.
60
+ A single column tour from spawn through both legs exceeds
61
+ 2200 ticks; parallel columns with W1 finishing first
62
+ (squad A delivers, squad B holds short of W2 until W1 latches)
63
+ makes it. Out-of-order (B arrives at W2 first) never satisfies
64
+ waypoint_sequence's W1→W2 latch.
65
+ overrides:
66
+ actors:
67
+ - {type: 2tnk, owner: agent, position: [5, 8], count: 3}
68
+ - {type: 1tnk, owner: agent, position: [7, 32], count: 3}
69
+ win_condition:
70
+ all_of:
71
+ - waypoint_sequence:
72
+ id: coord-easy
73
+ n: 2
74
+ radius: 7
75
+ points:
76
+ - {x: 110, y: 8, label: "W1 north-east rendezvous"}
77
+ - {x: 110, y: 32, label: "W2 south-east rendezvous"}
78
+ - within_ticks: 2200
79
+ fail_condition:
80
+ any_of:
81
+ - after_ticks: 2201
82
+ - not: {own_units_gte: 1}
83
+ max_turns: 40
84
+
85
+ medium:
86
+ description: >
87
+ Three waypoints in sequence: W1 NE, W2 bottom-left
88
+ (south-west detour), W3 SE — each n>=2, all within 3000
89
+ ticks. The detour leg through the south-west adds tour cost a
90
+ single column cannot absorb. Enemy pickets contest the two
91
+ eastern lanes. Loss cap 2.
92
+ overrides:
93
+ actors:
94
+ - {type: 2tnk, owner: agent, position: [5, 6], count: 3}
95
+ - {type: 1tnk, owner: agent, position: [7, 10], count: 3}
96
+ - {type: jeep, owner: agent, position: [5, 14], count: 2}
97
+ - {type: e1, owner: enemy, position: [55, 8], stance: 2, count: 2}
98
+ - {type: e1, owner: enemy, position: [55, 32], stance: 2, count: 2}
99
+ win_condition:
100
+ all_of:
101
+ - waypoint_sequence:
102
+ id: coord-medium
103
+ n: 2
104
+ radius: 7
105
+ points:
106
+ - {x: 115, y: 6, label: "W1 north-east"}
107
+ - {x: 20, y: 36, label: "W2 south-west detour"}
108
+ - {x: 115, y: 34, label: "W3 south-east"}
109
+ - within_ticks: 3000
110
+ - units_lost_lte: 2
111
+ fail_condition:
112
+ any_of:
113
+ - after_ticks: 3001
114
+ - not: {own_units_gte: 1}
115
+ - not: {units_lost_lte: 2}
116
+ max_turns: 48
117
+
118
+ hard:
119
+ description: >
120
+ Four waypoints in sequence with a strict ordering, 3200-tick
121
+ deadline, loss cap 1. Even with three squads, no single
122
+ column can absorb a full tour; the ordering forces W1 to
123
+ finish before W2 can be credited, etc., so squads must
124
+ stagger their arrivals naturally. Failure isolates *ordered*
125
+ parallel coordination — both simultaneous-only and serial
126
+ strategies miss.
127
+ overrides:
128
+ actors:
129
+ - {type: 2tnk, owner: agent, position: [5, 6], count: 3}
130
+ - {type: 1tnk, owner: agent, position: [7, 10], count: 3}
131
+ - {type: jeep, owner: agent, position: [5, 14], count: 3}
132
+ - {type: e1, owner: enemy, position: [55, 8], stance: 2, count: 2}
133
+ - {type: e3, owner: enemy, position: [55, 32], stance: 2, count: 2}
134
+ win_condition:
135
+ all_of:
136
+ - waypoint_sequence:
137
+ id: coord-hard
138
+ n: 2
139
+ radius: 6
140
+ points:
141
+ - {x: 115, y: 6, label: "W1 north-east"}
142
+ - {x: 20, y: 36, label: "W2 south-west"}
143
+ - {x: 115, y: 34, label: "W3 south-east"}
144
+ - {x: 60, y: 20, label: "W4 mid-east handoff"}
145
+ - within_ticks: 3200
146
+ - units_lost_lte: 1
147
+ fail_condition:
148
+ any_of:
149
+ - after_ticks: 3201
150
+ - not: {own_units_gte: 1}
151
+ - not: {units_lost_lte: 1}
152
+ max_turns: 52
openra_bench/scenarios/packs/perception-count-the-threat-small-k.yaml ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Validate: python -m openra_bench.scenarios.validate \
2
+ # openra_bench/scenarios/packs/perception-count-the-threat.yaml
3
+ # See ../CONTRIBUTING.md for the full win-condition grammar.
4
+ #
5
+ # PERCEPTION focus, EXACT-COUNT variant: the win is set to the actual
6
+ # hidden enemy count K. The model must scout enough to see all K — no
7
+ # more, no less — within a tight clock. Under-scouting fails the
8
+ # enemies_discovered_gte bar; over-scouting (sweeping the whole map)
9
+ # blows the clock or the units_lost_lte cap because reaching the
10
+ # trailing decoys costs ticks/casualties. The model has to REASON
11
+ # about WHERE the K-th enemy could be hiding from the partial scout
12
+ # so far, not just expand fog area-greedily.
13
+
14
+ meta:
15
+ id: perception-count-the-threat-small-k
16
+ title: 'Count the Threat (small-K) — Discover Exactly K Hidden Enemy Squads'
17
+ capability: perception
18
+ real_world_meaning: >
19
+ State-estimation under partial observability with a precise count
20
+ target: identify the exact number of hidden entities by inferring
21
+ where the remaining ones must be, given what has already been
22
+ cleared, before the clock runs out and without burning the force
23
+ on irrelevant ground.
24
+ robotics_analogue: >
25
+ Sensor-budgeted search for a known target count: locate all K
26
+ survivors / contraband / contacts in a bounded area, where missing
27
+ one fails the mission and over-searching exhausts the energy
28
+ budget on already-cleared zones.
29
+ benchmark_anchor:
30
+ - "ERQA state-estimation under partial observability"
31
+ - "POMDP exact-count sub-tasks"
32
+ - "ScienceWorld inventory before-commit"
33
+ - "ICU triage exact census"
34
+ author: yiyu-tian
35
+ # Companion to `perception-count-the-threat` (Wave 1 A7): both test
36
+ # the same cell (exact-K perception) but along different stress axes.
37
+ # A7 uses K=5 fixed medium / K seed-varied 4..7 hard with 2 spawn
38
+ # groups (count discrimination + spawn robustness). This pack uses
39
+ # K=2/3/4 with psychologically non-obvious hidden positions (the
40
+ # "where could the K-th be hiding?" inference axis).
41
+
42
+ base_map: rush-hour-arena
43
+
44
+ base:
45
+ agent: {faction: allies}
46
+ enemy: {faction: soviet}
47
+ tools: [move_units, attack_unit, stop_units]
48
+ planning: true
49
+ termination: {max_ticks: 12000}
50
+ actors:
51
+ - {type: jeep, owner: agent, position: [6, 5], count: 3}
52
+ - {type: 1tnk, owner: agent, position: [8, 8], count: 2}
53
+
54
+ levels:
55
+ # Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout
56
+ # fail actually fires within the run loop (eval_core.run_level
57
+ # iterates max_turns, not termination.max_ticks — without this
58
+ # alignment the after_ticks clause is unreachable and unmet wins
59
+ # exit as draw, not loss; see CLAUDE.md).
60
+ easy:
61
+ description: >
62
+ Exactly 2 enemy squads hidden in fog (one near-east, one
63
+ far-east). Find both within 2200 ticks losing <=1 unit. Both
64
+ are roughly on the same lane — a straight east sweep finds
65
+ them. Missing one or losing too many = loss.
66
+ overrides:
67
+ actors:
68
+ - {type: jeep, owner: agent, position: [6, 5], count: 3}
69
+ - {type: 1tnk, owner: agent, position: [8, 8], count: 2}
70
+ # Two distinct enemy squads, deliberately count K=2.
71
+ - {type: e1, owner: enemy, position: [55, 15], stance: 2, count: 2}
72
+ - {type: e3, owner: enemy, position: [105, 18], stance: 2, count: 2}
73
+ win_condition:
74
+ all_of:
75
+ - enemies_discovered_gte: 2
76
+ - within_ticks: 2200
77
+ - units_lost_lte: 1
78
+ fail_condition:
79
+ any_of:
80
+ - after_ticks: 2201
81
+ - not: {own_units_gte: 1}
82
+ - not: {units_lost_lte: 1}
83
+ max_turns: 26 # ceiling 93 + 90·25 = 2343 > 2201 ✓
84
+
85
+ medium:
86
+ description: >
87
+ Exactly 3 enemy squads scattered in three different directions
88
+ (north-east, south-east, and a non-obvious mid-north pocket).
89
+ Find all 3 within 2800 ticks losing <=2. A naive east-only
90
+ sweep finds 2 of 3 and runs out the clock searching the wrong
91
+ half of the map; the model must read where the remaining
92
+ unexplored mass is after each contact and commit there.
93
+ overrides:
94
+ actors:
95
+ - {type: jeep, owner: agent, position: [6, 5], count: 3}
96
+ - {type: 1tnk, owner: agent, position: [8, 8], count: 2}
97
+ - {type: e1, owner: enemy, position: [105, 8], stance: 2, count: 2} # NE
98
+ - {type: e3, owner: enemy, position: [105, 32], stance: 2, count: 2} # SE
99
+ - {type: dog, owner: enemy, position: [55, 4], stance: 2, count: 2} # mid-N
100
+ win_condition:
101
+ all_of:
102
+ - enemies_discovered_gte: 3
103
+ - within_ticks: 2800
104
+ - units_lost_lte: 2
105
+ fail_condition:
106
+ any_of:
107
+ - after_ticks: 2801
108
+ - not: {own_units_gte: 1}
109
+ max_turns: 32 # ceiling 2883 > 2801 ✓
110
+
111
+ hard:
112
+ description: >
113
+ Exactly 4 enemy squads. Two are "obvious" (along the natural
114
+ east-bound axes the spawn pushes toward), one is in the far-NW
115
+ strip behind the spawn, and one is in a tight south-of-mid
116
+ pocket. Loss cap is 2, clock is 3500. A best-effort sweep
117
+ finds 2–3; only inferring from which fog masses remain after
118
+ each contact identifies all 4 in time. Over-scouting (visiting
119
+ every remaining fog cell) blows the clock.
120
+ overrides:
121
+ actors:
122
+ - {type: jeep, owner: agent, position: [6, 5], count: 3}
123
+ - {type: 1tnk, owner: agent, position: [8, 8], count: 2}
124
+ - {type: e1, owner: enemy, position: [105, 8], stance: 2, count: 2}
125
+ - {type: e3, owner: enemy, position: [105, 32], stance: 2, count: 2}
126
+ - {type: dog, owner: enemy, position: [115, 4], stance: 2, count: 2} # far-NW relative to push
127
+ - {type: e2, owner: enemy, position: [60, 28], stance: 2, count: 2} # south-of-mid pocket
128
+ win_condition:
129
+ all_of:
130
+ - enemies_discovered_gte: 4
131
+ - within_ticks: 3500
132
+ - units_lost_lte: 2
133
+ fail_condition:
134
+ any_of:
135
+ - after_ticks: 3501
136
+ - not: {own_units_gte: 1}
137
+ max_turns: 40 # ceiling 3603 > 3501 ✓
openra_bench/scenarios/packs/risk-blockade-bypass.yaml ADDED
@@ -0,0 +1,151 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Validate: python -m openra_bench.scenarios.validate \
2
+ # openra_bench/scenarios/packs/risk-blockade-bypass.yaml
3
+ # See ../CONTRIBUTING.md for the full win-condition grammar.
4
+ #
5
+ # REASONING / RISK focus. The objective region is reachable via two
6
+ # axes: (a) the SHORT central corridor, blocked by a heavy enemy
7
+ # garrison that out-trades the agent's force; (b) a LONGER but
8
+ # lightly-defended detour. With a units_lost_lte cap and a clock
9
+ # that is tight but not too tight, the dominant strategy is the
10
+ # detour. Brute-forcing the corridor reduces the force below the
11
+ # attrition cap before reaching the objective.
12
+ #
13
+ # Difficulty escalates the cost of the wrong call: easy allows a
14
+ # slim head-on (loss <=4); medium tightens the cap; hard introduces
15
+ # a smaller forward force AND a tempo gate (the objective only
16
+ # scores after tick t0, so committing too early through the corridor
17
+ # and arriving wounded fails the after_ticks-then-attrition combo).
18
+
19
+ meta:
20
+ id: risk-blockade-bypass
21
+ title: 'Blockade Bypass — Reach the Objective by the Survivable Detour'
22
+ capability: reasoning
23
+ real_world_meaning: >
24
+ Route-risk assessment with a visibly hard direct path and a
25
+ longer survivable alternative under a binding attrition cap.
26
+ The cap punishes brute force; the clock punishes infinite
27
+ detour. The decision is committing the force to the route that
28
+ actually makes the deadline alive.
29
+ robotics_analogue: >
30
+ Field-robot mission routing rejecting a hazardous shortest path
31
+ for a survivable longer one within a time window — but the
32
+ longer path is not free, so the model must weigh attrition vs
33
+ deadline rather than always preferring the safe option.
34
+ benchmark_anchor:
35
+ - "MicroRTS route-risk / hazardous-corridor sub-tasks"
36
+ - "PlanBench cost-vs-survival route planning"
37
+ - "field-robotics: contested vs detour route under SLA"
38
+ - "convoy routing with attrition budget"
39
+ author: yiyu-tian
40
+ # Distinct from `reasoning-risk-route` (single-path hazard vs
41
+ # detour). This pack adds the explicit corridor-vs-detour
42
+ # bifurcation with an attrition cap binding tight enough that
43
+ # head-on brute-force fails the cap before reaching the goal.
44
+
45
+ base_map: rush-hour-arena
46
+
47
+ base:
48
+ agent:
49
+ faction: allies
50
+ cash: 0
51
+ enemy:
52
+ faction: soviet
53
+ cash: 0
54
+ bot_type: ''
55
+ interrupts:
56
+ enemy_unit_spotted: true
57
+ tools: [move_units, attack_unit, stop_units]
58
+ planning: true
59
+ termination: {max_ticks: 14000}
60
+ actors:
61
+ # Agent force, mid-west.
62
+ - {type: 2tnk, owner: agent, position: [8, 20], stance: 3, count: 3}
63
+ - {type: 1tnk, owner: agent, position: [8, 22], stance: 3, count: 3}
64
+ - {type: jeep, owner: agent, position: [6, 18], stance: 3, count: 2}
65
+ # Heavy CORRIDOR garrison (the wrong way): trades well, expensive
66
+ # to clear. Combination of e3 (rockets) + defensive gun.
67
+ - {type: e3, owner: enemy, position: [55, 20], stance: 2, count: 4}
68
+ - {type: gun, owner: enemy, position: [57, 22], stance: 3, count: 1}
69
+ - {type: e1, owner: enemy, position: [55, 22], stance: 2, count: 3}
70
+ # Light DETOUR garrison (the right way, top-N): one infantry
71
+ # picket; clears at low cost.
72
+ - {type: e1, owner: enemy, position: [55, 5], stance: 2, count: 2}
73
+ # Objective region marker: a static enemy building in the east.
74
+ - {type: fact, owner: enemy, position: [110, 20]}
75
+
76
+ # Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout fail
77
+ # fires within the run loop (eval_core.run_level iterates max_turns,
78
+ # not termination.max_ticks). The capability axis is route-risk under
79
+ # attrition cap; difficulty ladders on (clock, cap, force size) only —
80
+ # no orthogonal mechanic (e.g. tempo gate) introduced on hard, per
81
+ # CLAUDE.md "one new controlled variable per tier".
82
+ levels:
83
+ easy:
84
+ description: >
85
+ Reach the objective region near (110, 20) within 3500 ticks
86
+ losing <=4 units. The central corridor garrison (heavy at
87
+ (55, 20–22)) out-trades you 1-for-1; pushing through is
88
+ possible but spends the whole loss budget and may run the
89
+ clock. The northern detour (light picket at (55, 5)) costs
90
+ ~600 extra ticks but only ~1 unit. Decision is route choice.
91
+ overrides: {}
92
+ win_condition:
93
+ all_of:
94
+ - reach_region: {x: 110, y: 20, radius: 6}
95
+ - within_ticks: 3500
96
+ - units_lost_lte: 4
97
+ fail_condition:
98
+ any_of:
99
+ - after_ticks: 3501
100
+ - not: {own_units_gte: 1}
101
+ - not: {units_lost_lte: 4}
102
+ max_turns: 40 # ceiling 93 + 90·39 = 3603 > 3501 ✓
103
+
104
+ medium:
105
+ description: >
106
+ Same map, tighter cap (<=2 lost), 2800-tick deadline. The
107
+ corridor route is now infeasible — losses against the heavy
108
+ garrison exceed the cap deterministically. The detour is the
109
+ only path with positive margin, but it must be committed
110
+ promptly; dithering near the corridor before bouncing burns
111
+ ticks.
112
+ overrides: {}
113
+ win_condition:
114
+ all_of:
115
+ - reach_region: {x: 110, y: 20, radius: 6}
116
+ - within_ticks: 2800
117
+ - units_lost_lte: 2
118
+ fail_condition:
119
+ any_of:
120
+ - after_ticks: 2801
121
+ - not: {own_units_gte: 1}
122
+ - not: {units_lost_lte: 2}
123
+ max_turns: 32 # ceiling 2883 > 2801 ✓
124
+
125
+ hard:
126
+ description: >
127
+ Smaller agent force, attrition cap <=1, deadline 2400 ticks.
128
+ Pushing the corridor with the reduced force fails the cap
129
+ deterministically; the detour is the only path with positive
130
+ margin and it must be executed cleanly inside the budget.
131
+ overrides:
132
+ actors:
133
+ - {type: 2tnk, owner: agent, position: [8, 20], stance: 3, count: 2}
134
+ - {type: 1tnk, owner: agent, position: [8, 22], stance: 3, count: 2}
135
+ - {type: jeep, owner: agent, position: [6, 18], stance: 3, count: 2}
136
+ - {type: e3, owner: enemy, position: [55, 20], stance: 2, count: 4}
137
+ - {type: gun, owner: enemy, position: [57, 22], stance: 3, count: 1}
138
+ - {type: e1, owner: enemy, position: [55, 22], stance: 2, count: 3}
139
+ - {type: e1, owner: enemy, position: [55, 5], stance: 2, count: 2}
140
+ - {type: fact, owner: enemy, position: [110, 20]}
141
+ win_condition:
142
+ all_of:
143
+ - reach_region: {x: 110, y: 20, radius: 6}
144
+ - within_ticks: 2400
145
+ - units_lost_lte: 1
146
+ fail_condition:
147
+ any_of:
148
+ - after_ticks: 2401
149
+ - not: {own_units_gte: 1}
150
+ - not: {units_lost_lte: 1}
151
+ max_turns: 28 # ceiling 2523 > 2401 ✓
openra_bench/scenarios/packs/tempo-strike-window.yaml ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Validate: python -m openra_bench.scenarios.validate \
2
+ # openra_bench/scenarios/packs/tempo-strike-window.yaml
3
+ # See ../CONTRIBUTING.md for the full win-condition grammar.
4
+ #
5
+ # TEMPO focus, FORCED-LULL strike window. The agent must NOT engage
6
+ # before tick t0 (a "lull" / staging window) and then deliver the
7
+ # kill quota within a tight follow-up band before tick t1. Engaging
8
+ # early uses up the units against unfavourable spacing; waiting
9
+ # until t0 then committing is the dominant strategy.
10
+ #
11
+ # This is the cat-c11 canonical tempo design (after_ticks + kill bar
12
+ # + within_ticks), the one the audit said the suite still needs.
13
+ # Same axis across easy/medium/hard — only the lull length and the
14
+ # follow-up window tighten.
15
+ #
16
+ # Honestly NOT a true double-window (P2-3 happened-before predicate
17
+ # would be required to express "kill N1 in band 1 AND kill N2 in
18
+ # band 2 with a forced gap"). What this CAN express with current
19
+ # predicates: a single strike window after a forced wait, with
20
+ # attrition pressure.
21
+
22
+ meta:
23
+ id: tempo-strike-window
24
+ title: 'Strike Window — Wait for the Lull, Then Hit the Quota'
25
+ capability: reasoning
26
+ real_world_meaning: >
27
+ Timing discipline in a single strike window: do not engage
28
+ before the lull elapses, then deliver a kill quota inside the
29
+ follow-up band while staying under the attrition cap. Premature
30
+ engagement squanders the force; late engagement misses the
31
+ deadline.
32
+ robotics_analogue: >
33
+ Pulsed-mission timing: a UAV/team must hold station through a
34
+ no-action interval (e.g. avoiding detection / waiting for a
35
+ handoff window) before committing to its sortie, then complete
36
+ the engagement inside a hard window.
37
+ benchmark_anchor:
38
+ - "TextStarCraft II tempo / timed strike"
39
+ - "OSWorld scheduled-task chains"
40
+ - "PlanBench temporally-extended goals with hold-then-act"
41
+ - "pulsed-mission timing (UAV / fire-mission window)"
42
+ author: yiyu-tian
43
+ # Companion to `tempo-double-window` (Wave 1 A6). A6 is double-band
44
+ # (strike → lull-with-tsla → second strike). This pack is single-
45
+ # band with a forced pre-lull (no kills allowed before the gate)
46
+ # and a tight follow-up window — the cat-c11 canonical tempo design.
47
+
48
+ base_map: rush-hour-arena
49
+
50
+ base:
51
+ agent:
52
+ faction: allies
53
+ cash: 0
54
+ enemy:
55
+ faction: soviet
56
+ cash: 0
57
+ bot_type: ''
58
+ interrupts:
59
+ enemy_unit_spotted: true
60
+ own_unit_destroyed: true
61
+ tools: [move_units, attack_unit, stop_units]
62
+ planning: true
63
+ termination: {max_ticks: 12000}
64
+ actors:
65
+ - {type: 2tnk, owner: agent, position: [6, 18], stance: 3, count: 3}
66
+ - {type: 1tnk, owner: agent, position: [6, 22], stance: 3, count: 3}
67
+ - {type: jeep, owner: agent, position: [6, 14], stance: 3, count: 2}
68
+ # Enemy cluster positioned so the dominant strike is one
69
+ # coordinated push after the lull.
70
+ - {type: e1, owner: enemy, position: [40, 20], stance: 3, count: 4}
71
+ - {type: e3, owner: enemy, position: [42, 20], stance: 3, count: 2}
72
+
73
+ levels:
74
+ # Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout
75
+ # fail fires within the run loop (eval_core.run_level iterates
76
+ # max_turns, not termination.max_ticks).
77
+ easy:
78
+ description: >
79
+ Hold station, then strike. Do not have >=2 kills before tick
80
+ 2000 (early-engagement fail). After tick 2000, deliver >=4
81
+ kills before tick 4200, losing <=3. The lull is long enough
82
+ that pre-positioning is rewarded; the follow-up band is loose.
83
+ win_condition:
84
+ all_of:
85
+ - after_ticks: 2000
86
+ - units_killed_gte: 4
87
+ - within_ticks: 4200
88
+ - units_lost_lte: 3
89
+ fail_condition:
90
+ any_of:
91
+ - after_ticks: 4201
92
+ - not: {own_units_gte: 1}
93
+ - not: {units_lost_lte: 3}
94
+ # premature engagement: more than 1 kill before tick 2000
95
+ # is taken as "did not wait for the lull"
96
+ - all_of:
97
+ - units_killed_gte: 2
98
+ - within_ticks: 1999
99
+ max_turns: 48 # ceiling 93 + 90·47 = 4323 > 4201 ✓
100
+
101
+ medium:
102
+ description: >
103
+ Tighter pacing: no kills before tick 1800, then >=5 kills
104
+ before tick 3600, losing <=2. The lull is shorter and the
105
+ follow-up window narrower — pre-positioning during the lull
106
+ is required, not optional.
107
+ win_condition:
108
+ all_of:
109
+ - after_ticks: 1800
110
+ - units_killed_gte: 5
111
+ - within_ticks: 3600
112
+ - units_lost_lte: 2
113
+ fail_condition:
114
+ any_of:
115
+ - after_ticks: 3601
116
+ - not: {own_units_gte: 1}
117
+ - not: {units_lost_lte: 2}
118
+ - all_of:
119
+ - units_killed_gte: 1
120
+ - within_ticks: 1799
121
+ max_turns: 52
122
+
123
+ hard:
124
+ description: >
125
+ Aggressive lull-then-burst: no engagement at all before tick
126
+ 1500 (any kill = premature-fail), then >=6 kills before tick
127
+ 3000, losing <=1. The post-lull band is only 1500 ticks wide;
128
+ only an agent that has used the lull to assemble its force
129
+ at a launch point can hit the kill bar inside the window.
130
+ win_condition:
131
+ all_of:
132
+ - after_ticks: 1500
133
+ - units_killed_gte: 6
134
+ - within_ticks: 3000
135
+ - units_lost_lte: 1
136
+ fail_condition:
137
+ any_of:
138
+ - after_ticks: 3001
139
+ - not: {own_units_gte: 1}
140
+ - not: {units_lost_lte: 1}
141
+ - all_of:
142
+ - units_killed_gte: 1
143
+ - within_ticks: 1499
144
+ max_turns: 56
tests/test_coordination_ordered_rendezvous.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """coordination-ordered-rendezvous: ordered multi-waypoint delivery.
2
+
3
+ Structural tests verify the win predicate uses waypoint_sequence (not
4
+ parallel units_in_region clauses) so order is enforced — and that the
5
+ overall deadline binds. A pure-predicate unit test on waypoint_sequence
6
+ also confirms the order-violation case (later waypoint visited first
7
+ does NOT satisfy the sequence).
8
+ """
9
+ from __future__ import annotations
10
+
11
+ from pathlib import Path
12
+
13
+ import pytest
14
+
15
+ # openra_bench.scenarios eagerly imports the Rust adapter at module
16
+ # load (schema.py:15), so collection fails without the wheel. Skip the
17
+ # whole module if the env is missing — matches test_building_planning.
18
+ pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
19
+
20
+ from openra_bench.scenarios import load_pack
21
+ from openra_bench.scenarios.loader import compile_level
22
+ from openra_bench.scenarios.win_conditions import WinContext, evaluate
23
+
24
+ PACK = (
25
+ Path(__file__).parent.parent
26
+ / "openra_bench"
27
+ / "scenarios"
28
+ / "packs"
29
+ / "coordination-ordered-rendezvous.yaml"
30
+ )
31
+
32
+ EXPECTED_WAYPOINTS = {"easy": 2, "medium": 3, "hard": 4}
33
+
34
+
35
+ def _win_clauses(c):
36
+ return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
37
+
38
+
39
+ def _fail_clauses(c):
40
+ return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
41
+
42
+
43
+ def _seq_value(c):
44
+ for cl in _win_clauses(c):
45
+ if "waypoint_sequence" in cl:
46
+ return cl["waypoint_sequence"]
47
+ return None
48
+
49
+
50
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
51
+ def test_level_uses_waypoint_sequence_not_simultaneous_regions(level):
52
+ pack = load_pack(PACK)
53
+ assert pack.meta.capability == "action"
54
+ c = compile_level(pack, level)
55
+
56
+ win = _win_clauses(c)
57
+ seq = _seq_value(c)
58
+ assert seq is not None, f"{level}: must use waypoint_sequence for order"
59
+ assert seq["n"] == 2, f"{level}: each leg requires >=2 units"
60
+ assert len(seq["points"]) == EXPECTED_WAYPOINTS[level], (
61
+ f"{level}: expected {EXPECTED_WAYPOINTS[level]} waypoints, got {len(seq['points'])}"
62
+ )
63
+ # critically NOT using simultaneous units_in_region (the
64
+ # action-multiunit-coordination shape) — that doesn't enforce order
65
+ assert not any("units_in_region_gte" in cl for cl in win), (
66
+ f"{level}: should use waypoint_sequence, not units_in_region_gte"
67
+ )
68
+
69
+ # deadline binds
70
+ wt = [cl["within_ticks"] for cl in win if "within_ticks" in cl][0]
71
+ assert wt < c.max_turns * 90, f"{level}: within_ticks {wt} doesn't bind"
72
+
73
+ # every level can LOSE
74
+ fail = _fail_clauses(c)
75
+ assert any("after_ticks" in cl for cl in fail), (
76
+ f"{level}: missing timeout in fail_condition"
77
+ )
78
+
79
+
80
+ def test_waypoint_count_scales_with_difficulty():
81
+ pack = load_pack(PACK)
82
+ counts = []
83
+ for level in ("easy", "medium", "hard"):
84
+ c = compile_level(pack, level)
85
+ counts.append(len(_seq_value(c)["points"]))
86
+ assert counts == [2, 3, 4], f"waypoint count should ladder 2->3->4, got {counts}"
87
+
88
+
89
+ # ---- pure-Python predicate unit tests on waypoint_sequence ordering ----
90
+
91
+
92
+ class _FakeSignals:
93
+ def __init__(self):
94
+ self.game_tick = 100
95
+ self.seq_progress: dict = {}
96
+
97
+
98
+ def _ctx(units):
99
+ return WinContext(signals=_FakeSignals(), render_state={"units_summary": units})
100
+
101
+
102
+ def test_waypoint_sequence_advances_only_in_order():
103
+ sig = _FakeSignals()
104
+ pts = [{"x": 110, "y": 8}, {"x": 110, "y": 32}]
105
+ spec = {"waypoint_sequence": {"id": "t", "n": 2, "radius": 7, "points": pts}}
106
+
107
+ # Visit W2 first — must NOT satisfy
108
+ ctx = WinContext(
109
+ signals=sig,
110
+ render_state={"units_summary": [
111
+ {"cell_x": 110, "cell_y": 32}, {"cell_x": 111, "cell_y": 32},
112
+ ]},
113
+ )
114
+ assert evaluate(spec, ctx) is False
115
+ # idx must still be 0 (no premature advance from W2-only presence)
116
+ assert sig.seq_progress.get("t", 0) == 0
117
+
118
+ # Now visit W1
119
+ ctx = WinContext(
120
+ signals=sig,
121
+ render_state={"units_summary": [
122
+ {"cell_x": 110, "cell_y": 8}, {"cell_x": 111, "cell_y": 8},
123
+ ]},
124
+ )
125
+ assert evaluate(spec, ctx) is False # W1 advances; W2 not yet
126
+ assert sig.seq_progress["t"] == 1
127
+
128
+ # Then visit W2 — sequence completes
129
+ ctx = WinContext(
130
+ signals=sig,
131
+ render_state={"units_summary": [
132
+ {"cell_x": 110, "cell_y": 32}, {"cell_x": 111, "cell_y": 32},
133
+ ]},
134
+ )
135
+ assert evaluate(spec, ctx) is True
136
+ assert sig.seq_progress["t"] == 2
137
+
138
+
139
+ def test_waypoint_sequence_requires_min_n_units():
140
+ sig = _FakeSignals()
141
+ pts = [{"x": 110, "y": 8}, {"x": 110, "y": 32}]
142
+ spec = {"waypoint_sequence": {"id": "t2", "n": 2, "radius": 7, "points": pts}}
143
+
144
+ # Only 1 unit at W1 — does not advance
145
+ ctx = WinContext(
146
+ signals=sig,
147
+ render_state={"units_summary": [{"cell_x": 110, "cell_y": 8}]},
148
+ )
149
+ assert evaluate(spec, ctx) is False
150
+ assert sig.seq_progress.get("t2", 0) == 0
151
+
152
+
153
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
154
+ def test_pack_runs_and_donothing_loses(level):
155
+ from openra_bench.eval_core import run_level
156
+
157
+ c = compile_level(load_pack(PACK), level)
158
+ assert c.map_supported
159
+ res = run_level(c, lambda rs, C: [C.observe()], seed=1)
160
+ assert res.outcome == "loss", (
161
+ f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
162
+ )
tests/test_hard_tier.py CHANGED
@@ -82,6 +82,22 @@ NOT_APPLICABLE = {
82
  "instead of handed coordinates). Adding seed-driven spawn variation "
83
  "would introduce a second uncontrolled variable and break the clean "
84
  "medium→hard attribution.",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  }
86
 
87
  # No-adversary maps: spawn variation applies but a force-loss
 
82
  "instead of handed coordinates). Adding seed-driven spawn variation "
83
  "would introduce a second uncontrolled variable and break the clean "
84
  "medium→hard attribution.",
85
+ # PR #8 from yiyu-tian: 4 packs that tighten one decision axis per
86
+ # tier rather than per-spawn variation. UPGRADED would force a
87
+ # second variable per tier and break the clean attribution model.
88
+ "perception-count-the-threat-small-k": "hard tightens K (2→3→4) "
89
+ "and the positional non-obviousness of each squad — adding spawn "
90
+ "variation would dilute the count-and-infer signal. Companion to "
91
+ "perception-count-the-threat (UPGRADED above), which IS spawn-varied.",
92
+ "coordination-ordered-rendezvous": "hard tightens the waypoint "
93
+ "count (2→3→4) and the overall deadline; spawn variation would "
94
+ "compete with the order-enforcement axis for attribution.",
95
+ "tempo-strike-window": "hard tightens the lull length, follow-up "
96
+ "window, and attrition cap (cat-c11 canonical tempo design). "
97
+ "Spawn variation would dilute the tempo-discipline signal.",
98
+ "risk-blockade-bypass": "hard adds a tempo gate (after_ticks) and "
99
+ "shrinks the attrition cap; spawn variation would muddy the "
100
+ "corridor-vs-detour route-choice signal.",
101
  }
102
 
103
  # No-adversary maps: spawn variation applies but a force-loss
tests/test_perception_count_the_threat_small_k.py ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """perception-count-the-threat: exact-K enemy discovery under fog.
2
+
3
+ Structural tests assert the predicate shape — each level discriminates
4
+ by the exact K (enemies_discovered_gte) and has a real fail_condition.
5
+ Run-smoke test confirms the pack loads and a do-nothing agent loses on
6
+ the timeout (proves fail_condition is wired through compile()).
7
+
8
+ Module-level importorskip because openra_bench.scenarios imports the
9
+ Rust adapter at import time (schema.py:15), so even pure-Python
10
+ structural tests need the wheel to load the package. Same pattern as
11
+ test_building_planning.py.
12
+ """
13
+ from __future__ import annotations
14
+
15
+ from pathlib import Path
16
+
17
+ import pytest
18
+
19
+ pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
20
+
21
+ from openra_bench.scenarios import load_pack
22
+ from openra_bench.scenarios.loader import compile_level
23
+
24
+ PACK = (
25
+ Path(__file__).parent.parent
26
+ / "openra_bench"
27
+ / "scenarios"
28
+ / "packs"
29
+ / "perception-count-the-threat-small-k.yaml"
30
+ )
31
+
32
+ EXPECTED_K = {"easy": 2, "medium": 3, "hard": 4}
33
+
34
+
35
+ def _win_clauses(c):
36
+ return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
37
+
38
+
39
+ def _fail_clauses(c):
40
+ return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
41
+
42
+
43
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
44
+ def test_level_predicates_isolate_exact_count(level):
45
+ pack = load_pack(PACK)
46
+ assert pack.meta.capability == "perception"
47
+ c = compile_level(pack, level)
48
+
49
+ win = _win_clauses(c)
50
+ # exact K target for this level
51
+ ks = [cl["enemies_discovered_gte"] for cl in win if "enemies_discovered_gte" in cl]
52
+ assert ks == [EXPECTED_K[level]], (
53
+ f"{level}: expected enemies_discovered_gte={EXPECTED_K[level]}, got {ks}"
54
+ )
55
+ # tight clock binds (< max_turns * 90)
56
+ wt = [cl["within_ticks"] for cl in win if "within_ticks" in cl][0]
57
+ assert wt < c.max_turns * 90, f"{level}: within_ticks {wt} doesn't bind"
58
+ # attrition cap present
59
+ assert any("units_lost_lte" in cl for cl in win), (
60
+ f"{level}: missing units_lost_lte attrition cap"
61
+ )
62
+
63
+ # every level can LOSE (timeout-fail at min)
64
+ fail = _fail_clauses(c)
65
+ assert any("after_ticks" in cl for cl in fail), (
66
+ f"{level}: missing timeout in fail_condition"
67
+ )
68
+
69
+
70
+ def test_k_increases_monotonically_with_difficulty():
71
+ pack = load_pack(PACK)
72
+ ks = []
73
+ for level in ("easy", "medium", "hard"):
74
+ c = compile_level(pack, level)
75
+ ks.append([cl["enemies_discovered_gte"] for cl in _win_clauses(c) if "enemies_discovered_gte" in cl][0])
76
+ assert ks == sorted(ks) and ks[0] < ks[-1], (
77
+ f"K must scale with difficulty; got {ks}"
78
+ )
79
+
80
+
81
+ def test_hidden_actor_count_matches_K():
82
+ """K hidden enemies must actually exist on the map — otherwise the
83
+ win is unsatisfiable."""
84
+ pack = load_pack(PACK)
85
+ for level in ("easy", "medium", "hard"):
86
+ c = compile_level(pack, level)
87
+ enemy_actors = [a for a in c.scenario.actors if a.owner == "enemy"]
88
+ # count distinct enemy *positions* (each group is a discoverable squad)
89
+ positions = {(a.position[0], a.position[1]) for a in enemy_actors}
90
+ assert len(positions) >= EXPECTED_K[level], (
91
+ f"{level}: only {len(positions)} enemy positions on map, "
92
+ f"win needs {EXPECTED_K[level]}"
93
+ )
94
+
95
+
96
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
97
+ def test_pack_runs_and_donothing_loses(level):
98
+ from openra_bench.eval_core import run_level
99
+
100
+ c = compile_level(load_pack(PACK), level)
101
+ assert c.map_supported
102
+ res = run_level(c, lambda rs, C: [C.observe()], seed=1)
103
+ # do-nothing must lose on the timeout (proves fail_condition is live)
104
+ assert res.outcome == "loss", (
105
+ f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
106
+ )
tests/test_risk_blockade_bypass.py ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """risk-blockade-bypass: route-risk decision with two visible routes
2
+ (heavy corridor vs longer detour).
3
+
4
+ Structural tests verify the win predicate combines a positional
5
+ objective (reach_region) with a binding attrition cap and clock —
6
+ the three pieces that make this a risk decision, not a navigation
7
+ or combat test. Hard adds a tempo gate (after_ticks) tightening the
8
+ risk calculus (can't speedrun).
9
+ """
10
+ from __future__ import annotations
11
+
12
+ from pathlib import Path
13
+
14
+ import pytest
15
+
16
+ # openra_bench.scenarios eagerly imports the Rust adapter at module
17
+ # load (schema.py:15), so collection fails without the wheel. Skip the
18
+ # whole module if the env is missing — matches test_building_planning.
19
+ pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
20
+
21
+ from openra_bench.scenarios import load_pack
22
+ from openra_bench.scenarios.loader import compile_level
23
+
24
+ PACK = (
25
+ Path(__file__).parent.parent
26
+ / "openra_bench"
27
+ / "scenarios"
28
+ / "packs"
29
+ / "risk-blockade-bypass.yaml"
30
+ )
31
+
32
+ # (deadline, loss_cap, after_ticks_gate_or_None)
33
+ # Tempo gate dropped from hard to keep the difficulty axis on route-
34
+ # risk only (CLAUDE.md C.10: one new controlled variable per tier).
35
+ EXPECTED = {
36
+ "easy": (3500, 4, None),
37
+ "medium": (2800, 2, None),
38
+ "hard": (2400, 1, None),
39
+ }
40
+
41
+
42
+ def _win_clauses(c):
43
+ return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
44
+
45
+
46
+ def _fail_clauses(c):
47
+ return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
48
+
49
+
50
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
51
+ def test_level_encodes_reach_with_attrition_and_clock(level):
52
+ pack = load_pack(PACK)
53
+ assert pack.meta.capability == "reasoning"
54
+ c = compile_level(pack, level)
55
+
56
+ win = _win_clauses(c)
57
+ reach = [cl["reach_region"] for cl in win if "reach_region" in cl]
58
+ deadline = [cl["within_ticks"] for cl in win if "within_ticks" in cl]
59
+ loss_cap = [cl["units_lost_lte"] for cl in win if "units_lost_lte" in cl]
60
+ after = [cl["after_ticks"] for cl in win if "after_ticks" in cl]
61
+
62
+ assert len(reach) == 1, f"{level}: must reach exactly one region"
63
+ assert reach[0]["x"] == 110, f"{level}: objective should be at the east marker"
64
+ assert len(deadline) == 1 and len(loss_cap) == 1, (
65
+ f"{level}: must have both within_ticks and units_lost_lte"
66
+ )
67
+
68
+ exp_deadline, exp_loss, exp_after = EXPECTED[level]
69
+ assert deadline[0] == exp_deadline
70
+ assert loss_cap[0] == exp_loss
71
+ if exp_after is None:
72
+ assert after == [], f"{level}: should not have an after_ticks gate"
73
+ else:
74
+ assert after == [exp_after], f"{level}: expected after_ticks={exp_after}, got {after}"
75
+
76
+
77
+ def test_attrition_cap_tightens_with_difficulty():
78
+ caps = [EXPECTED[lv][1] for lv in ("easy", "medium", "hard")]
79
+ assert caps == [4, 2, 1] and caps == sorted(caps, reverse=True), (
80
+ f"attrition cap must tighten 4->2->1: {caps}"
81
+ )
82
+
83
+
84
+ def test_deadline_tightens_with_difficulty():
85
+ deadlines = [EXPECTED[lv][0] for lv in ("easy", "medium", "hard")]
86
+ assert deadlines == sorted(deadlines, reverse=True), (
87
+ f"deadline must tighten: {deadlines}"
88
+ )
89
+
90
+
91
+ def test_both_routes_visibly_present_on_map():
92
+ """The map must contain BOTH the heavy-corridor garrison (mid-east)
93
+ and the light-detour picket (north) — otherwise it isn't a route
94
+ choice scenario."""
95
+ pack = load_pack(PACK)
96
+ for level in ("easy", "medium", "hard"):
97
+ c = compile_level(pack, level)
98
+ enemy = [a for a in c.scenario.actors if a.owner == "enemy"]
99
+ # heavy corridor garrison at y~20 (mid-latitude)
100
+ corridor = [a for a in enemy if 15 <= a.position[1] <= 25 and 40 <= a.position[0] <= 70]
101
+ # light detour picket at y~5 (top)
102
+ detour = [a for a in enemy if a.position[1] <= 10 and 40 <= a.position[0] <= 70]
103
+ assert corridor, f"{level}: missing heavy corridor garrison"
104
+ assert detour, f"{level}: missing light detour picket"
105
+
106
+
107
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
108
+ def test_pack_runs_and_donothing_loses(level):
109
+ from openra_bench.eval_core import run_level
110
+
111
+ c = compile_level(load_pack(PACK), level)
112
+ assert c.map_supported
113
+ res = run_level(c, lambda rs, C: [C.observe()], seed=1)
114
+ assert res.outcome == "loss", (
115
+ f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
116
+ )
tests/test_tempo_strike_window.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """tempo-strike-window: forced lull then kill quota.
2
+
3
+ Structural tests verify every level encodes (a) an after_ticks lull
4
+ gate, (b) a kill quota, (c) a within_ticks deadline, (d) an attrition
5
+ cap, AND (e) a "premature engagement" fail clause keyed off
6
+ units_killed_gte inside the lull window.
7
+ """
8
+ from __future__ import annotations
9
+
10
+ from pathlib import Path
11
+
12
+ import pytest
13
+
14
+ # openra_bench.scenarios eagerly imports the Rust adapter at module
15
+ # load (schema.py:15), so collection fails without the wheel. Skip the
16
+ # whole module if the env is missing — matches test_building_planning.
17
+ pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
18
+
19
+ from openra_bench.scenarios import load_pack
20
+ from openra_bench.scenarios.loader import compile_level
21
+
22
+ PACK = (
23
+ Path(__file__).parent.parent
24
+ / "openra_bench"
25
+ / "scenarios"
26
+ / "packs"
27
+ / "tempo-strike-window.yaml"
28
+ )
29
+
30
+ # (lull_until, kill_target, deadline, loss_cap)
31
+ EXPECTED = {
32
+ "easy": (2000, 4, 4200, 3),
33
+ "medium": (1800, 5, 3600, 2),
34
+ "hard": (1500, 6, 3000, 1),
35
+ }
36
+
37
+
38
+ def _win_clauses(c):
39
+ return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
40
+
41
+
42
+ def _fail_clauses(c):
43
+ return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
44
+
45
+
46
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
47
+ def test_level_encodes_lull_kill_deadline_attrition(level):
48
+ pack = load_pack(PACK)
49
+ assert pack.meta.capability == "reasoning"
50
+ c = compile_level(pack, level)
51
+
52
+ win = _win_clauses(c)
53
+ after = [cl["after_ticks"] for cl in win if "after_ticks" in cl]
54
+ killed = [cl["units_killed_gte"] for cl in win if "units_killed_gte" in cl]
55
+ within = [cl["within_ticks"] for cl in win if "within_ticks" in cl]
56
+ lost = [cl["units_lost_lte"] for cl in win if "units_lost_lte" in cl]
57
+
58
+ assert len(after) == 1 and len(killed) == 1 and len(within) == 1 and len(lost) == 1, (
59
+ f"{level}: must have exactly one of each (after_ticks/units_killed_gte/"
60
+ f"within_ticks/units_lost_lte)"
61
+ )
62
+ lull_until, kill_target, deadline, loss_cap = EXPECTED[level]
63
+ assert after[0] == lull_until, f"{level}: lull_until expected {lull_until}, got {after[0]}"
64
+ assert killed[0] == kill_target, f"{level}: kill_target expected {kill_target}, got {killed[0]}"
65
+ assert within[0] == deadline, f"{level}: deadline expected {deadline}, got {within[0]}"
66
+ assert lost[0] == loss_cap, f"{level}: loss_cap expected {loss_cap}, got {lost[0]}"
67
+ assert after[0] < within[0], f"{level}: lull must end before deadline"
68
+
69
+
70
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
71
+ def test_fail_condition_includes_premature_engagement(level):
72
+ """The 'premature engagement' fail = units_killed_gte:N AND
73
+ within_ticks:lull-1 — i.e. having too many kills before the lull
74
+ elapses immediately loses."""
75
+ pack = load_pack(PACK)
76
+ c = compile_level(pack, level)
77
+ fail = _fail_clauses(c)
78
+ lull_until = EXPECTED[level][0]
79
+
80
+ has_premature = False
81
+ for cl in fail:
82
+ if "all_of" in cl:
83
+ inner = cl["all_of"]
84
+ has_killed = any("units_killed_gte" in x for x in inner)
85
+ within_cap = [x["within_ticks"] for x in inner if "within_ticks" in x]
86
+ if has_killed and within_cap and within_cap[0] < lull_until:
87
+ has_premature = True
88
+ break
89
+ assert has_premature, (
90
+ f"{level}: missing premature-engagement fail "
91
+ f"(units_killed_gte AND within_ticks < {lull_until})"
92
+ )
93
+
94
+
95
+ def test_difficulty_tightens_lull_kill_deadline_attrition():
96
+ """easy -> medium -> hard: shorter lull, higher kill bar, shorter
97
+ deadline, tighter attrition cap."""
98
+ lulls = [EXPECTED[lv][0] for lv in ("easy", "medium", "hard")]
99
+ kills = [EXPECTED[lv][1] for lv in ("easy", "medium", "hard")]
100
+ deadlines = [EXPECTED[lv][2] for lv in ("easy", "medium", "hard")]
101
+ losses = [EXPECTED[lv][3] for lv in ("easy", "medium", "hard")]
102
+
103
+ assert lulls == sorted(lulls, reverse=True), f"lull should shrink: {lulls}"
104
+ assert kills == sorted(kills), f"kill target should grow: {kills}"
105
+ assert deadlines == sorted(deadlines, reverse=True), f"deadline should shrink: {deadlines}"
106
+ assert losses == sorted(losses, reverse=True), f"loss cap should shrink: {losses}"
107
+
108
+
109
+ @pytest.mark.parametrize("level", ["easy", "medium", "hard"])
110
+ def test_pack_runs_and_donothing_loses(level):
111
+ from openra_bench.eval_core import run_level
112
+
113
+ c = compile_level(load_pack(PACK), level)
114
+ assert c.map_supported
115
+ res = run_level(c, lambda rs, C: [C.observe()], seed=1)
116
+ assert res.outcome == "loss", (
117
+ f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
118
+ )