Spaces:
Running
feat(scenarios): merge PR #8 from yiyu-tian — 4 new packs (rebased onto Wave 1+2)
Browse filesCherry-picks the 4 scenario packs from PR #8 onto current main (the PR
was based on commit 1400f05 pre-Wave 1; merging as-is would have
deleted the 17 Wave 1+2 packs and 4 engine PRs).
New packs (each adds a transfer-anchored bench cell):
- perception-count-the-threat-small-k (renamed from -the-threat to
avoid collision with Wave 1 A7). Different stress axis from A7:
K=2/3/4 with psychologically non-obvious hidden positions vs A7's
K=4..7 seeded with 2 spawn groups. Both ship; complementary.
Anchor: ERQA / POMDP / ScienceWorld / ICU triage.
- coordination-ordered-rendezvous. Complements A5 staggered-window:
this one tests visit ORDER via waypoint_sequence (n=2) across 2/3/4
waypoints. Anchor: Watch-And-Help / SMAC / PERT / supply-chain
ordered handoff.
- tempo-strike-window. Complements A6 double-window: this one is the
single-band canonical cat-c11 tempo design (forced lull then kill
quota in follow-up window). Anchor: TextStarCraft II / OSWorld /
PlanBench temporal goals.
- risk-blockade-bypass. New cell (no overlap with reasoning-risk-
route): heavy corridor vs light detour with attrition cap making
brute-force infeasible. Anchor: MicroRTS route-risk / PlanBench
cost-vs-survival / field-robotics convoy routing.
Per-pack additions vs PR #8 source:
- meta.benchmark_anchor: list[str] populated (the new schema
requirement we added in Wave 2 foundation, commit 17f024b — post
the PR's base).
- All 4 packs added to tests/test_hard_tier.py::NOT_APPLICABLE with
documented reasons (none use spawn_point variation on hard; each
tightens one decision axis per tier instead).
- perception-count-the-threat-small-k renamed (lowercase 'k' to
satisfy the slug validator; meta.id, test file, and references
all updated).
- author: yiyu-tian on each (credit preserved).
NOT cherry-picked from PR #8:
- The "importorskip on 18 pre-existing test files" infrastructure
commit (cbf068c) — touches files unrelated to the scenarios; the
local suite passes without it (719 pre-PR, 759 post-merge).
Test suite: 719 → 759 passing (+40 — 4 new packs × ~10 tests each).
Closes #8.
- openra_bench/scenarios/packs/coordination-ordered-rendezvous.yaml +152 -0
- openra_bench/scenarios/packs/perception-count-the-threat-small-k.yaml +137 -0
- openra_bench/scenarios/packs/risk-blockade-bypass.yaml +151 -0
- openra_bench/scenarios/packs/tempo-strike-window.yaml +144 -0
- tests/test_coordination_ordered_rendezvous.py +162 -0
- tests/test_hard_tier.py +16 -0
- tests/test_perception_count_the_threat_small_k.py +106 -0
- tests/test_risk_blockade_bypass.py +116 -0
- tests/test_tempo_strike_window.py +118 -0
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Validate: python -m openra_bench.scenarios.validate \
|
| 2 |
+
# openra_bench/scenarios/packs/coordination-staggered-window.yaml
|
| 3 |
+
# See ../CONTRIBUTING.md for the full win-condition grammar.
|
| 4 |
+
#
|
| 5 |
+
# ACTION focus, ORDERED parallel rendezvous. Two/three squads must
|
| 6 |
+
# deliver >=2 units to a sequence of regions IN ORDER (W1 before W2
|
| 7 |
+
# before W3), within a tight overall deadline. Order is enforced by
|
| 8 |
+
# `waypoint_sequence` (stateful latch: W2 only counts after W1 was
|
| 9 |
+
# satisfied at some prior tick). The deadline is tight enough that
|
| 10 |
+
# a single column cannot tour all waypoints in time — parallel
|
| 11 |
+
# columns are required AND the deposit at W1 must come first.
|
| 12 |
+
#
|
| 13 |
+
# Differs from action-multiunit-coordination (simultaneous delivery
|
| 14 |
+
# to N regions at one deadline) on the ORDER axis: this scenario
|
| 15 |
+
# fails if W2 is reached before W1, even if both end up satisfied.
|
| 16 |
+
# Same axis across easy/medium/hard — only the number of waypoints
|
| 17 |
+
# and the deadline tighten.
|
| 18 |
+
|
| 19 |
+
meta:
|
| 20 |
+
id: coordination-ordered-rendezvous
|
| 21 |
+
title: 'Ordered Rendezvous — Deliver Squads to Waypoints In Sequence'
|
| 22 |
+
capability: action
|
| 23 |
+
real_world_meaning: >
|
| 24 |
+
Parallel multi-team delivery under an ordering constraint:
|
| 25 |
+
waypoint A must be reached before waypoint B before waypoint C,
|
| 26 |
+
each requiring two or more units present, all within one tight
|
| 27 |
+
overall window. Skipping or out-of-order delivery does not
|
| 28 |
+
count. Single-column tours blow the deadline; only concurrent
|
| 29 |
+
dispatch with leg-by-leg ordering succeeds.
|
| 30 |
+
robotics_analogue: >
|
| 31 |
+
Sequenced multi-depot fleet dispatch with a precedence
|
| 32 |
+
constraint: team A must arrive at depot 1 before team B may be
|
| 33 |
+
credited for arriving at depot 2 (e.g. handoff / supply chain),
|
| 34 |
+
and the overall window does not permit a single team to do all
|
| 35 |
+
legs in series.
|
| 36 |
+
benchmark_anchor:
|
| 37 |
+
- "Watch-And-Help concurrent multi-agent with sequenced sub-tasks"
|
| 38 |
+
- "SMAC asymmetric squad sequencing"
|
| 39 |
+
- "PERT/CPM precedence-constrained dispatch"
|
| 40 |
+
- "supply-chain ordered handoff: depot A before depot B"
|
| 41 |
+
author: yiyu-tian
|
| 42 |
+
|
| 43 |
+
base_map: rush-hour-arena
|
| 44 |
+
|
| 45 |
+
base:
|
| 46 |
+
agent: {faction: allies}
|
| 47 |
+
enemy: {faction: soviet}
|
| 48 |
+
tools: [move_units, attack_unit, stop_units]
|
| 49 |
+
planning: true
|
| 50 |
+
termination: {max_ticks: 6000}
|
| 51 |
+
actors:
|
| 52 |
+
- {type: 2tnk, owner: agent, position: [5, 8], count: 3}
|
| 53 |
+
- {type: 1tnk, owner: agent, position: [7, 32], count: 3}
|
| 54 |
+
|
| 55 |
+
levels:
|
| 56 |
+
easy:
|
| 57 |
+
description: >
|
| 58 |
+
Two waypoints in sequence: W1 NE (110, 8) before W2 SE
|
| 59 |
+
(110, 32), each requiring >=2 units, all within 2200 ticks.
|
| 60 |
+
A single column tour from spawn through both legs exceeds
|
| 61 |
+
2200 ticks; parallel columns with W1 finishing first
|
| 62 |
+
(squad A delivers, squad B holds short of W2 until W1 latches)
|
| 63 |
+
makes it. Out-of-order (B arrives at W2 first) never satisfies
|
| 64 |
+
waypoint_sequence's W1→W2 latch.
|
| 65 |
+
overrides:
|
| 66 |
+
actors:
|
| 67 |
+
- {type: 2tnk, owner: agent, position: [5, 8], count: 3}
|
| 68 |
+
- {type: 1tnk, owner: agent, position: [7, 32], count: 3}
|
| 69 |
+
win_condition:
|
| 70 |
+
all_of:
|
| 71 |
+
- waypoint_sequence:
|
| 72 |
+
id: coord-easy
|
| 73 |
+
n: 2
|
| 74 |
+
radius: 7
|
| 75 |
+
points:
|
| 76 |
+
- {x: 110, y: 8, label: "W1 north-east rendezvous"}
|
| 77 |
+
- {x: 110, y: 32, label: "W2 south-east rendezvous"}
|
| 78 |
+
- within_ticks: 2200
|
| 79 |
+
fail_condition:
|
| 80 |
+
any_of:
|
| 81 |
+
- after_ticks: 2201
|
| 82 |
+
- not: {own_units_gte: 1}
|
| 83 |
+
max_turns: 40
|
| 84 |
+
|
| 85 |
+
medium:
|
| 86 |
+
description: >
|
| 87 |
+
Three waypoints in sequence: W1 NE, W2 bottom-left
|
| 88 |
+
(south-west detour), W3 SE — each n>=2, all within 3000
|
| 89 |
+
ticks. The detour leg through the south-west adds tour cost a
|
| 90 |
+
single column cannot absorb. Enemy pickets contest the two
|
| 91 |
+
eastern lanes. Loss cap 2.
|
| 92 |
+
overrides:
|
| 93 |
+
actors:
|
| 94 |
+
- {type: 2tnk, owner: agent, position: [5, 6], count: 3}
|
| 95 |
+
- {type: 1tnk, owner: agent, position: [7, 10], count: 3}
|
| 96 |
+
- {type: jeep, owner: agent, position: [5, 14], count: 2}
|
| 97 |
+
- {type: e1, owner: enemy, position: [55, 8], stance: 2, count: 2}
|
| 98 |
+
- {type: e1, owner: enemy, position: [55, 32], stance: 2, count: 2}
|
| 99 |
+
win_condition:
|
| 100 |
+
all_of:
|
| 101 |
+
- waypoint_sequence:
|
| 102 |
+
id: coord-medium
|
| 103 |
+
n: 2
|
| 104 |
+
radius: 7
|
| 105 |
+
points:
|
| 106 |
+
- {x: 115, y: 6, label: "W1 north-east"}
|
| 107 |
+
- {x: 20, y: 36, label: "W2 south-west detour"}
|
| 108 |
+
- {x: 115, y: 34, label: "W3 south-east"}
|
| 109 |
+
- within_ticks: 3000
|
| 110 |
+
- units_lost_lte: 2
|
| 111 |
+
fail_condition:
|
| 112 |
+
any_of:
|
| 113 |
+
- after_ticks: 3001
|
| 114 |
+
- not: {own_units_gte: 1}
|
| 115 |
+
- not: {units_lost_lte: 2}
|
| 116 |
+
max_turns: 48
|
| 117 |
+
|
| 118 |
+
hard:
|
| 119 |
+
description: >
|
| 120 |
+
Four waypoints in sequence with a strict ordering, 3200-tick
|
| 121 |
+
deadline, loss cap 1. Even with three squads, no single
|
| 122 |
+
column can absorb a full tour; the ordering forces W1 to
|
| 123 |
+
finish before W2 can be credited, etc., so squads must
|
| 124 |
+
stagger their arrivals naturally. Failure isolates *ordered*
|
| 125 |
+
parallel coordination — both simultaneous-only and serial
|
| 126 |
+
strategies miss.
|
| 127 |
+
overrides:
|
| 128 |
+
actors:
|
| 129 |
+
- {type: 2tnk, owner: agent, position: [5, 6], count: 3}
|
| 130 |
+
- {type: 1tnk, owner: agent, position: [7, 10], count: 3}
|
| 131 |
+
- {type: jeep, owner: agent, position: [5, 14], count: 3}
|
| 132 |
+
- {type: e1, owner: enemy, position: [55, 8], stance: 2, count: 2}
|
| 133 |
+
- {type: e3, owner: enemy, position: [55, 32], stance: 2, count: 2}
|
| 134 |
+
win_condition:
|
| 135 |
+
all_of:
|
| 136 |
+
- waypoint_sequence:
|
| 137 |
+
id: coord-hard
|
| 138 |
+
n: 2
|
| 139 |
+
radius: 6
|
| 140 |
+
points:
|
| 141 |
+
- {x: 115, y: 6, label: "W1 north-east"}
|
| 142 |
+
- {x: 20, y: 36, label: "W2 south-west"}
|
| 143 |
+
- {x: 115, y: 34, label: "W3 south-east"}
|
| 144 |
+
- {x: 60, y: 20, label: "W4 mid-east handoff"}
|
| 145 |
+
- within_ticks: 3200
|
| 146 |
+
- units_lost_lte: 1
|
| 147 |
+
fail_condition:
|
| 148 |
+
any_of:
|
| 149 |
+
- after_ticks: 3201
|
| 150 |
+
- not: {own_units_gte: 1}
|
| 151 |
+
- not: {units_lost_lte: 1}
|
| 152 |
+
max_turns: 52
|
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Validate: python -m openra_bench.scenarios.validate \
|
| 2 |
+
# openra_bench/scenarios/packs/perception-count-the-threat.yaml
|
| 3 |
+
# See ../CONTRIBUTING.md for the full win-condition grammar.
|
| 4 |
+
#
|
| 5 |
+
# PERCEPTION focus, EXACT-COUNT variant: the win is set to the actual
|
| 6 |
+
# hidden enemy count K. The model must scout enough to see all K — no
|
| 7 |
+
# more, no less — within a tight clock. Under-scouting fails the
|
| 8 |
+
# enemies_discovered_gte bar; over-scouting (sweeping the whole map)
|
| 9 |
+
# blows the clock or the units_lost_lte cap because reaching the
|
| 10 |
+
# trailing decoys costs ticks/casualties. The model has to REASON
|
| 11 |
+
# about WHERE the K-th enemy could be hiding from the partial scout
|
| 12 |
+
# so far, not just expand fog area-greedily.
|
| 13 |
+
|
| 14 |
+
meta:
|
| 15 |
+
id: perception-count-the-threat-small-k
|
| 16 |
+
title: 'Count the Threat (small-K) — Discover Exactly K Hidden Enemy Squads'
|
| 17 |
+
capability: perception
|
| 18 |
+
real_world_meaning: >
|
| 19 |
+
State-estimation under partial observability with a precise count
|
| 20 |
+
target: identify the exact number of hidden entities by inferring
|
| 21 |
+
where the remaining ones must be, given what has already been
|
| 22 |
+
cleared, before the clock runs out and without burning the force
|
| 23 |
+
on irrelevant ground.
|
| 24 |
+
robotics_analogue: >
|
| 25 |
+
Sensor-budgeted search for a known target count: locate all K
|
| 26 |
+
survivors / contraband / contacts in a bounded area, where missing
|
| 27 |
+
one fails the mission and over-searching exhausts the energy
|
| 28 |
+
budget on already-cleared zones.
|
| 29 |
+
benchmark_anchor:
|
| 30 |
+
- "ERQA state-estimation under partial observability"
|
| 31 |
+
- "POMDP exact-count sub-tasks"
|
| 32 |
+
- "ScienceWorld inventory before-commit"
|
| 33 |
+
- "ICU triage exact census"
|
| 34 |
+
author: yiyu-tian
|
| 35 |
+
# Companion to `perception-count-the-threat` (Wave 1 A7): both test
|
| 36 |
+
# the same cell (exact-K perception) but along different stress axes.
|
| 37 |
+
# A7 uses K=5 fixed medium / K seed-varied 4..7 hard with 2 spawn
|
| 38 |
+
# groups (count discrimination + spawn robustness). This pack uses
|
| 39 |
+
# K=2/3/4 with psychologically non-obvious hidden positions (the
|
| 40 |
+
# "where could the K-th be hiding?" inference axis).
|
| 41 |
+
|
| 42 |
+
base_map: rush-hour-arena
|
| 43 |
+
|
| 44 |
+
base:
|
| 45 |
+
agent: {faction: allies}
|
| 46 |
+
enemy: {faction: soviet}
|
| 47 |
+
tools: [move_units, attack_unit, stop_units]
|
| 48 |
+
planning: true
|
| 49 |
+
termination: {max_ticks: 12000}
|
| 50 |
+
actors:
|
| 51 |
+
- {type: jeep, owner: agent, position: [6, 5], count: 3}
|
| 52 |
+
- {type: 1tnk, owner: agent, position: [8, 8], count: 2}
|
| 53 |
+
|
| 54 |
+
levels:
|
| 55 |
+
# Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout
|
| 56 |
+
# fail actually fires within the run loop (eval_core.run_level
|
| 57 |
+
# iterates max_turns, not termination.max_ticks — without this
|
| 58 |
+
# alignment the after_ticks clause is unreachable and unmet wins
|
| 59 |
+
# exit as draw, not loss; see CLAUDE.md).
|
| 60 |
+
easy:
|
| 61 |
+
description: >
|
| 62 |
+
Exactly 2 enemy squads hidden in fog (one near-east, one
|
| 63 |
+
far-east). Find both within 2200 ticks losing <=1 unit. Both
|
| 64 |
+
are roughly on the same lane — a straight east sweep finds
|
| 65 |
+
them. Missing one or losing too many = loss.
|
| 66 |
+
overrides:
|
| 67 |
+
actors:
|
| 68 |
+
- {type: jeep, owner: agent, position: [6, 5], count: 3}
|
| 69 |
+
- {type: 1tnk, owner: agent, position: [8, 8], count: 2}
|
| 70 |
+
# Two distinct enemy squads, deliberately count K=2.
|
| 71 |
+
- {type: e1, owner: enemy, position: [55, 15], stance: 2, count: 2}
|
| 72 |
+
- {type: e3, owner: enemy, position: [105, 18], stance: 2, count: 2}
|
| 73 |
+
win_condition:
|
| 74 |
+
all_of:
|
| 75 |
+
- enemies_discovered_gte: 2
|
| 76 |
+
- within_ticks: 2200
|
| 77 |
+
- units_lost_lte: 1
|
| 78 |
+
fail_condition:
|
| 79 |
+
any_of:
|
| 80 |
+
- after_ticks: 2201
|
| 81 |
+
- not: {own_units_gte: 1}
|
| 82 |
+
- not: {units_lost_lte: 1}
|
| 83 |
+
max_turns: 26 # ceiling 93 + 90·25 = 2343 > 2201 ✓
|
| 84 |
+
|
| 85 |
+
medium:
|
| 86 |
+
description: >
|
| 87 |
+
Exactly 3 enemy squads scattered in three different directions
|
| 88 |
+
(north-east, south-east, and a non-obvious mid-north pocket).
|
| 89 |
+
Find all 3 within 2800 ticks losing <=2. A naive east-only
|
| 90 |
+
sweep finds 2 of 3 and runs out the clock searching the wrong
|
| 91 |
+
half of the map; the model must read where the remaining
|
| 92 |
+
unexplored mass is after each contact and commit there.
|
| 93 |
+
overrides:
|
| 94 |
+
actors:
|
| 95 |
+
- {type: jeep, owner: agent, position: [6, 5], count: 3}
|
| 96 |
+
- {type: 1tnk, owner: agent, position: [8, 8], count: 2}
|
| 97 |
+
- {type: e1, owner: enemy, position: [105, 8], stance: 2, count: 2} # NE
|
| 98 |
+
- {type: e3, owner: enemy, position: [105, 32], stance: 2, count: 2} # SE
|
| 99 |
+
- {type: dog, owner: enemy, position: [55, 4], stance: 2, count: 2} # mid-N
|
| 100 |
+
win_condition:
|
| 101 |
+
all_of:
|
| 102 |
+
- enemies_discovered_gte: 3
|
| 103 |
+
- within_ticks: 2800
|
| 104 |
+
- units_lost_lte: 2
|
| 105 |
+
fail_condition:
|
| 106 |
+
any_of:
|
| 107 |
+
- after_ticks: 2801
|
| 108 |
+
- not: {own_units_gte: 1}
|
| 109 |
+
max_turns: 32 # ceiling 2883 > 2801 ✓
|
| 110 |
+
|
| 111 |
+
hard:
|
| 112 |
+
description: >
|
| 113 |
+
Exactly 4 enemy squads. Two are "obvious" (along the natural
|
| 114 |
+
east-bound axes the spawn pushes toward), one is in the far-NW
|
| 115 |
+
strip behind the spawn, and one is in a tight south-of-mid
|
| 116 |
+
pocket. Loss cap is 2, clock is 3500. A best-effort sweep
|
| 117 |
+
finds 2–3; only inferring from which fog masses remain after
|
| 118 |
+
each contact identifies all 4 in time. Over-scouting (visiting
|
| 119 |
+
every remaining fog cell) blows the clock.
|
| 120 |
+
overrides:
|
| 121 |
+
actors:
|
| 122 |
+
- {type: jeep, owner: agent, position: [6, 5], count: 3}
|
| 123 |
+
- {type: 1tnk, owner: agent, position: [8, 8], count: 2}
|
| 124 |
+
- {type: e1, owner: enemy, position: [105, 8], stance: 2, count: 2}
|
| 125 |
+
- {type: e3, owner: enemy, position: [105, 32], stance: 2, count: 2}
|
| 126 |
+
- {type: dog, owner: enemy, position: [115, 4], stance: 2, count: 2} # far-NW relative to push
|
| 127 |
+
- {type: e2, owner: enemy, position: [60, 28], stance: 2, count: 2} # south-of-mid pocket
|
| 128 |
+
win_condition:
|
| 129 |
+
all_of:
|
| 130 |
+
- enemies_discovered_gte: 4
|
| 131 |
+
- within_ticks: 3500
|
| 132 |
+
- units_lost_lte: 2
|
| 133 |
+
fail_condition:
|
| 134 |
+
any_of:
|
| 135 |
+
- after_ticks: 3501
|
| 136 |
+
- not: {own_units_gte: 1}
|
| 137 |
+
max_turns: 40 # ceiling 3603 > 3501 ✓
|
|
@@ -0,0 +1,151 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Validate: python -m openra_bench.scenarios.validate \
|
| 2 |
+
# openra_bench/scenarios/packs/risk-blockade-bypass.yaml
|
| 3 |
+
# See ../CONTRIBUTING.md for the full win-condition grammar.
|
| 4 |
+
#
|
| 5 |
+
# REASONING / RISK focus. The objective region is reachable via two
|
| 6 |
+
# axes: (a) the SHORT central corridor, blocked by a heavy enemy
|
| 7 |
+
# garrison that out-trades the agent's force; (b) a LONGER but
|
| 8 |
+
# lightly-defended detour. With a units_lost_lte cap and a clock
|
| 9 |
+
# that is tight but not too tight, the dominant strategy is the
|
| 10 |
+
# detour. Brute-forcing the corridor reduces the force below the
|
| 11 |
+
# attrition cap before reaching the objective.
|
| 12 |
+
#
|
| 13 |
+
# Difficulty escalates the cost of the wrong call: easy allows a
|
| 14 |
+
# slim head-on (loss <=4); medium tightens the cap; hard introduces
|
| 15 |
+
# a smaller forward force AND a tempo gate (the objective only
|
| 16 |
+
# scores after tick t0, so committing too early through the corridor
|
| 17 |
+
# and arriving wounded fails the after_ticks-then-attrition combo).
|
| 18 |
+
|
| 19 |
+
meta:
|
| 20 |
+
id: risk-blockade-bypass
|
| 21 |
+
title: 'Blockade Bypass — Reach the Objective by the Survivable Detour'
|
| 22 |
+
capability: reasoning
|
| 23 |
+
real_world_meaning: >
|
| 24 |
+
Route-risk assessment with a visibly hard direct path and a
|
| 25 |
+
longer survivable alternative under a binding attrition cap.
|
| 26 |
+
The cap punishes brute force; the clock punishes infinite
|
| 27 |
+
detour. The decision is committing the force to the route that
|
| 28 |
+
actually makes the deadline alive.
|
| 29 |
+
robotics_analogue: >
|
| 30 |
+
Field-robot mission routing rejecting a hazardous shortest path
|
| 31 |
+
for a survivable longer one within a time window — but the
|
| 32 |
+
longer path is not free, so the model must weigh attrition vs
|
| 33 |
+
deadline rather than always preferring the safe option.
|
| 34 |
+
benchmark_anchor:
|
| 35 |
+
- "MicroRTS route-risk / hazardous-corridor sub-tasks"
|
| 36 |
+
- "PlanBench cost-vs-survival route planning"
|
| 37 |
+
- "field-robotics: contested vs detour route under SLA"
|
| 38 |
+
- "convoy routing with attrition budget"
|
| 39 |
+
author: yiyu-tian
|
| 40 |
+
# Distinct from `reasoning-risk-route` (single-path hazard vs
|
| 41 |
+
# detour). This pack adds the explicit corridor-vs-detour
|
| 42 |
+
# bifurcation with an attrition cap binding tight enough that
|
| 43 |
+
# head-on brute-force fails the cap before reaching the goal.
|
| 44 |
+
|
| 45 |
+
base_map: rush-hour-arena
|
| 46 |
+
|
| 47 |
+
base:
|
| 48 |
+
agent:
|
| 49 |
+
faction: allies
|
| 50 |
+
cash: 0
|
| 51 |
+
enemy:
|
| 52 |
+
faction: soviet
|
| 53 |
+
cash: 0
|
| 54 |
+
bot_type: ''
|
| 55 |
+
interrupts:
|
| 56 |
+
enemy_unit_spotted: true
|
| 57 |
+
tools: [move_units, attack_unit, stop_units]
|
| 58 |
+
planning: true
|
| 59 |
+
termination: {max_ticks: 14000}
|
| 60 |
+
actors:
|
| 61 |
+
# Agent force, mid-west.
|
| 62 |
+
- {type: 2tnk, owner: agent, position: [8, 20], stance: 3, count: 3}
|
| 63 |
+
- {type: 1tnk, owner: agent, position: [8, 22], stance: 3, count: 3}
|
| 64 |
+
- {type: jeep, owner: agent, position: [6, 18], stance: 3, count: 2}
|
| 65 |
+
# Heavy CORRIDOR garrison (the wrong way): trades well, expensive
|
| 66 |
+
# to clear. Combination of e3 (rockets) + defensive gun.
|
| 67 |
+
- {type: e3, owner: enemy, position: [55, 20], stance: 2, count: 4}
|
| 68 |
+
- {type: gun, owner: enemy, position: [57, 22], stance: 3, count: 1}
|
| 69 |
+
- {type: e1, owner: enemy, position: [55, 22], stance: 2, count: 3}
|
| 70 |
+
# Light DETOUR garrison (the right way, top-N): one infantry
|
| 71 |
+
# picket; clears at low cost.
|
| 72 |
+
- {type: e1, owner: enemy, position: [55, 5], stance: 2, count: 2}
|
| 73 |
+
# Objective region marker: a static enemy building in the east.
|
| 74 |
+
- {type: fact, owner: enemy, position: [110, 20]}
|
| 75 |
+
|
| 76 |
+
# Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout fail
|
| 77 |
+
# fires within the run loop (eval_core.run_level iterates max_turns,
|
| 78 |
+
# not termination.max_ticks). The capability axis is route-risk under
|
| 79 |
+
# attrition cap; difficulty ladders on (clock, cap, force size) only —
|
| 80 |
+
# no orthogonal mechanic (e.g. tempo gate) introduced on hard, per
|
| 81 |
+
# CLAUDE.md "one new controlled variable per tier".
|
| 82 |
+
levels:
|
| 83 |
+
easy:
|
| 84 |
+
description: >
|
| 85 |
+
Reach the objective region near (110, 20) within 3500 ticks
|
| 86 |
+
losing <=4 units. The central corridor garrison (heavy at
|
| 87 |
+
(55, 20–22)) out-trades you 1-for-1; pushing through is
|
| 88 |
+
possible but spends the whole loss budget and may run the
|
| 89 |
+
clock. The northern detour (light picket at (55, 5)) costs
|
| 90 |
+
~600 extra ticks but only ~1 unit. Decision is route choice.
|
| 91 |
+
overrides: {}
|
| 92 |
+
win_condition:
|
| 93 |
+
all_of:
|
| 94 |
+
- reach_region: {x: 110, y: 20, radius: 6}
|
| 95 |
+
- within_ticks: 3500
|
| 96 |
+
- units_lost_lte: 4
|
| 97 |
+
fail_condition:
|
| 98 |
+
any_of:
|
| 99 |
+
- after_ticks: 3501
|
| 100 |
+
- not: {own_units_gte: 1}
|
| 101 |
+
- not: {units_lost_lte: 4}
|
| 102 |
+
max_turns: 40 # ceiling 93 + 90·39 = 3603 > 3501 ✓
|
| 103 |
+
|
| 104 |
+
medium:
|
| 105 |
+
description: >
|
| 106 |
+
Same map, tighter cap (<=2 lost), 2800-tick deadline. The
|
| 107 |
+
corridor route is now infeasible — losses against the heavy
|
| 108 |
+
garrison exceed the cap deterministically. The detour is the
|
| 109 |
+
only path with positive margin, but it must be committed
|
| 110 |
+
promptly; dithering near the corridor before bouncing burns
|
| 111 |
+
ticks.
|
| 112 |
+
overrides: {}
|
| 113 |
+
win_condition:
|
| 114 |
+
all_of:
|
| 115 |
+
- reach_region: {x: 110, y: 20, radius: 6}
|
| 116 |
+
- within_ticks: 2800
|
| 117 |
+
- units_lost_lte: 2
|
| 118 |
+
fail_condition:
|
| 119 |
+
any_of:
|
| 120 |
+
- after_ticks: 2801
|
| 121 |
+
- not: {own_units_gte: 1}
|
| 122 |
+
- not: {units_lost_lte: 2}
|
| 123 |
+
max_turns: 32 # ceiling 2883 > 2801 ✓
|
| 124 |
+
|
| 125 |
+
hard:
|
| 126 |
+
description: >
|
| 127 |
+
Smaller agent force, attrition cap <=1, deadline 2400 ticks.
|
| 128 |
+
Pushing the corridor with the reduced force fails the cap
|
| 129 |
+
deterministically; the detour is the only path with positive
|
| 130 |
+
margin and it must be executed cleanly inside the budget.
|
| 131 |
+
overrides:
|
| 132 |
+
actors:
|
| 133 |
+
- {type: 2tnk, owner: agent, position: [8, 20], stance: 3, count: 2}
|
| 134 |
+
- {type: 1tnk, owner: agent, position: [8, 22], stance: 3, count: 2}
|
| 135 |
+
- {type: jeep, owner: agent, position: [6, 18], stance: 3, count: 2}
|
| 136 |
+
- {type: e3, owner: enemy, position: [55, 20], stance: 2, count: 4}
|
| 137 |
+
- {type: gun, owner: enemy, position: [57, 22], stance: 3, count: 1}
|
| 138 |
+
- {type: e1, owner: enemy, position: [55, 22], stance: 2, count: 3}
|
| 139 |
+
- {type: e1, owner: enemy, position: [55, 5], stance: 2, count: 2}
|
| 140 |
+
- {type: fact, owner: enemy, position: [110, 20]}
|
| 141 |
+
win_condition:
|
| 142 |
+
all_of:
|
| 143 |
+
- reach_region: {x: 110, y: 20, radius: 6}
|
| 144 |
+
- within_ticks: 2400
|
| 145 |
+
- units_lost_lte: 1
|
| 146 |
+
fail_condition:
|
| 147 |
+
any_of:
|
| 148 |
+
- after_ticks: 2401
|
| 149 |
+
- not: {own_units_gte: 1}
|
| 150 |
+
- not: {units_lost_lte: 1}
|
| 151 |
+
max_turns: 28 # ceiling 2523 > 2401 ✓
|
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Validate: python -m openra_bench.scenarios.validate \
|
| 2 |
+
# openra_bench/scenarios/packs/tempo-strike-window.yaml
|
| 3 |
+
# See ../CONTRIBUTING.md for the full win-condition grammar.
|
| 4 |
+
#
|
| 5 |
+
# TEMPO focus, FORCED-LULL strike window. The agent must NOT engage
|
| 6 |
+
# before tick t0 (a "lull" / staging window) and then deliver the
|
| 7 |
+
# kill quota within a tight follow-up band before tick t1. Engaging
|
| 8 |
+
# early uses up the units against unfavourable spacing; waiting
|
| 9 |
+
# until t0 then committing is the dominant strategy.
|
| 10 |
+
#
|
| 11 |
+
# This is the cat-c11 canonical tempo design (after_ticks + kill bar
|
| 12 |
+
# + within_ticks), the one the audit said the suite still needs.
|
| 13 |
+
# Same axis across easy/medium/hard — only the lull length and the
|
| 14 |
+
# follow-up window tighten.
|
| 15 |
+
#
|
| 16 |
+
# Honestly NOT a true double-window (P2-3 happened-before predicate
|
| 17 |
+
# would be required to express "kill N1 in band 1 AND kill N2 in
|
| 18 |
+
# band 2 with a forced gap"). What this CAN express with current
|
| 19 |
+
# predicates: a single strike window after a forced wait, with
|
| 20 |
+
# attrition pressure.
|
| 21 |
+
|
| 22 |
+
meta:
|
| 23 |
+
id: tempo-strike-window
|
| 24 |
+
title: 'Strike Window — Wait for the Lull, Then Hit the Quota'
|
| 25 |
+
capability: reasoning
|
| 26 |
+
real_world_meaning: >
|
| 27 |
+
Timing discipline in a single strike window: do not engage
|
| 28 |
+
before the lull elapses, then deliver a kill quota inside the
|
| 29 |
+
follow-up band while staying under the attrition cap. Premature
|
| 30 |
+
engagement squanders the force; late engagement misses the
|
| 31 |
+
deadline.
|
| 32 |
+
robotics_analogue: >
|
| 33 |
+
Pulsed-mission timing: a UAV/team must hold station through a
|
| 34 |
+
no-action interval (e.g. avoiding detection / waiting for a
|
| 35 |
+
handoff window) before committing to its sortie, then complete
|
| 36 |
+
the engagement inside a hard window.
|
| 37 |
+
benchmark_anchor:
|
| 38 |
+
- "TextStarCraft II tempo / timed strike"
|
| 39 |
+
- "OSWorld scheduled-task chains"
|
| 40 |
+
- "PlanBench temporally-extended goals with hold-then-act"
|
| 41 |
+
- "pulsed-mission timing (UAV / fire-mission window)"
|
| 42 |
+
author: yiyu-tian
|
| 43 |
+
# Companion to `tempo-double-window` (Wave 1 A6). A6 is double-band
|
| 44 |
+
# (strike → lull-with-tsla → second strike). This pack is single-
|
| 45 |
+
# band with a forced pre-lull (no kills allowed before the gate)
|
| 46 |
+
# and a tight follow-up window — the cat-c11 canonical tempo design.
|
| 47 |
+
|
| 48 |
+
base_map: rush-hour-arena
|
| 49 |
+
|
| 50 |
+
base:
|
| 51 |
+
agent:
|
| 52 |
+
faction: allies
|
| 53 |
+
cash: 0
|
| 54 |
+
enemy:
|
| 55 |
+
faction: soviet
|
| 56 |
+
cash: 0
|
| 57 |
+
bot_type: ''
|
| 58 |
+
interrupts:
|
| 59 |
+
enemy_unit_spotted: true
|
| 60 |
+
own_unit_destroyed: true
|
| 61 |
+
tools: [move_units, attack_unit, stop_units]
|
| 62 |
+
planning: true
|
| 63 |
+
termination: {max_ticks: 12000}
|
| 64 |
+
actors:
|
| 65 |
+
- {type: 2tnk, owner: agent, position: [6, 18], stance: 3, count: 3}
|
| 66 |
+
- {type: 1tnk, owner: agent, position: [6, 22], stance: 3, count: 3}
|
| 67 |
+
- {type: jeep, owner: agent, position: [6, 14], stance: 3, count: 2}
|
| 68 |
+
# Enemy cluster positioned so the dominant strike is one
|
| 69 |
+
# coordinated push after the lull.
|
| 70 |
+
- {type: e1, owner: enemy, position: [40, 20], stance: 3, count: 4}
|
| 71 |
+
- {type: e3, owner: enemy, position: [42, 20], stance: 3, count: 2}
|
| 72 |
+
|
| 73 |
+
levels:
|
| 74 |
+
# Tick budgets aligned to 93 + 90·(max_turns - 1) so the timeout
|
| 75 |
+
# fail fires within the run loop (eval_core.run_level iterates
|
| 76 |
+
# max_turns, not termination.max_ticks).
|
| 77 |
+
easy:
|
| 78 |
+
description: >
|
| 79 |
+
Hold station, then strike. Do not have >=2 kills before tick
|
| 80 |
+
2000 (early-engagement fail). After tick 2000, deliver >=4
|
| 81 |
+
kills before tick 4200, losing <=3. The lull is long enough
|
| 82 |
+
that pre-positioning is rewarded; the follow-up band is loose.
|
| 83 |
+
win_condition:
|
| 84 |
+
all_of:
|
| 85 |
+
- after_ticks: 2000
|
| 86 |
+
- units_killed_gte: 4
|
| 87 |
+
- within_ticks: 4200
|
| 88 |
+
- units_lost_lte: 3
|
| 89 |
+
fail_condition:
|
| 90 |
+
any_of:
|
| 91 |
+
- after_ticks: 4201
|
| 92 |
+
- not: {own_units_gte: 1}
|
| 93 |
+
- not: {units_lost_lte: 3}
|
| 94 |
+
# premature engagement: more than 1 kill before tick 2000
|
| 95 |
+
# is taken as "did not wait for the lull"
|
| 96 |
+
- all_of:
|
| 97 |
+
- units_killed_gte: 2
|
| 98 |
+
- within_ticks: 1999
|
| 99 |
+
max_turns: 48 # ceiling 93 + 90·47 = 4323 > 4201 ✓
|
| 100 |
+
|
| 101 |
+
medium:
|
| 102 |
+
description: >
|
| 103 |
+
Tighter pacing: no kills before tick 1800, then >=5 kills
|
| 104 |
+
before tick 3600, losing <=2. The lull is shorter and the
|
| 105 |
+
follow-up window narrower — pre-positioning during the lull
|
| 106 |
+
is required, not optional.
|
| 107 |
+
win_condition:
|
| 108 |
+
all_of:
|
| 109 |
+
- after_ticks: 1800
|
| 110 |
+
- units_killed_gte: 5
|
| 111 |
+
- within_ticks: 3600
|
| 112 |
+
- units_lost_lte: 2
|
| 113 |
+
fail_condition:
|
| 114 |
+
any_of:
|
| 115 |
+
- after_ticks: 3601
|
| 116 |
+
- not: {own_units_gte: 1}
|
| 117 |
+
- not: {units_lost_lte: 2}
|
| 118 |
+
- all_of:
|
| 119 |
+
- units_killed_gte: 1
|
| 120 |
+
- within_ticks: 1799
|
| 121 |
+
max_turns: 52
|
| 122 |
+
|
| 123 |
+
hard:
|
| 124 |
+
description: >
|
| 125 |
+
Aggressive lull-then-burst: no engagement at all before tick
|
| 126 |
+
1500 (any kill = premature-fail), then >=6 kills before tick
|
| 127 |
+
3000, losing <=1. The post-lull band is only 1500 ticks wide;
|
| 128 |
+
only an agent that has used the lull to assemble its force
|
| 129 |
+
at a launch point can hit the kill bar inside the window.
|
| 130 |
+
win_condition:
|
| 131 |
+
all_of:
|
| 132 |
+
- after_ticks: 1500
|
| 133 |
+
- units_killed_gte: 6
|
| 134 |
+
- within_ticks: 3000
|
| 135 |
+
- units_lost_lte: 1
|
| 136 |
+
fail_condition:
|
| 137 |
+
any_of:
|
| 138 |
+
- after_ticks: 3001
|
| 139 |
+
- not: {own_units_gte: 1}
|
| 140 |
+
- not: {units_lost_lte: 1}
|
| 141 |
+
- all_of:
|
| 142 |
+
- units_killed_gte: 1
|
| 143 |
+
- within_ticks: 1499
|
| 144 |
+
max_turns: 56
|
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""coordination-ordered-rendezvous: ordered multi-waypoint delivery.
|
| 2 |
+
|
| 3 |
+
Structural tests verify the win predicate uses waypoint_sequence (not
|
| 4 |
+
parallel units_in_region clauses) so order is enforced — and that the
|
| 5 |
+
overall deadline binds. A pure-predicate unit test on waypoint_sequence
|
| 6 |
+
also confirms the order-violation case (later waypoint visited first
|
| 7 |
+
does NOT satisfy the sequence).
|
| 8 |
+
"""
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
|
| 13 |
+
import pytest
|
| 14 |
+
|
| 15 |
+
# openra_bench.scenarios eagerly imports the Rust adapter at module
|
| 16 |
+
# load (schema.py:15), so collection fails without the wheel. Skip the
|
| 17 |
+
# whole module if the env is missing — matches test_building_planning.
|
| 18 |
+
pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
|
| 19 |
+
|
| 20 |
+
from openra_bench.scenarios import load_pack
|
| 21 |
+
from openra_bench.scenarios.loader import compile_level
|
| 22 |
+
from openra_bench.scenarios.win_conditions import WinContext, evaluate
|
| 23 |
+
|
| 24 |
+
PACK = (
|
| 25 |
+
Path(__file__).parent.parent
|
| 26 |
+
/ "openra_bench"
|
| 27 |
+
/ "scenarios"
|
| 28 |
+
/ "packs"
|
| 29 |
+
/ "coordination-ordered-rendezvous.yaml"
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
EXPECTED_WAYPOINTS = {"easy": 2, "medium": 3, "hard": 4}
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _win_clauses(c):
|
| 36 |
+
return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def _fail_clauses(c):
|
| 40 |
+
return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def _seq_value(c):
|
| 44 |
+
for cl in _win_clauses(c):
|
| 45 |
+
if "waypoint_sequence" in cl:
|
| 46 |
+
return cl["waypoint_sequence"]
|
| 47 |
+
return None
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 51 |
+
def test_level_uses_waypoint_sequence_not_simultaneous_regions(level):
|
| 52 |
+
pack = load_pack(PACK)
|
| 53 |
+
assert pack.meta.capability == "action"
|
| 54 |
+
c = compile_level(pack, level)
|
| 55 |
+
|
| 56 |
+
win = _win_clauses(c)
|
| 57 |
+
seq = _seq_value(c)
|
| 58 |
+
assert seq is not None, f"{level}: must use waypoint_sequence for order"
|
| 59 |
+
assert seq["n"] == 2, f"{level}: each leg requires >=2 units"
|
| 60 |
+
assert len(seq["points"]) == EXPECTED_WAYPOINTS[level], (
|
| 61 |
+
f"{level}: expected {EXPECTED_WAYPOINTS[level]} waypoints, got {len(seq['points'])}"
|
| 62 |
+
)
|
| 63 |
+
# critically NOT using simultaneous units_in_region (the
|
| 64 |
+
# action-multiunit-coordination shape) — that doesn't enforce order
|
| 65 |
+
assert not any("units_in_region_gte" in cl for cl in win), (
|
| 66 |
+
f"{level}: should use waypoint_sequence, not units_in_region_gte"
|
| 67 |
+
)
|
| 68 |
+
|
| 69 |
+
# deadline binds
|
| 70 |
+
wt = [cl["within_ticks"] for cl in win if "within_ticks" in cl][0]
|
| 71 |
+
assert wt < c.max_turns * 90, f"{level}: within_ticks {wt} doesn't bind"
|
| 72 |
+
|
| 73 |
+
# every level can LOSE
|
| 74 |
+
fail = _fail_clauses(c)
|
| 75 |
+
assert any("after_ticks" in cl for cl in fail), (
|
| 76 |
+
f"{level}: missing timeout in fail_condition"
|
| 77 |
+
)
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def test_waypoint_count_scales_with_difficulty():
|
| 81 |
+
pack = load_pack(PACK)
|
| 82 |
+
counts = []
|
| 83 |
+
for level in ("easy", "medium", "hard"):
|
| 84 |
+
c = compile_level(pack, level)
|
| 85 |
+
counts.append(len(_seq_value(c)["points"]))
|
| 86 |
+
assert counts == [2, 3, 4], f"waypoint count should ladder 2->3->4, got {counts}"
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ---- pure-Python predicate unit tests on waypoint_sequence ordering ----
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
class _FakeSignals:
|
| 93 |
+
def __init__(self):
|
| 94 |
+
self.game_tick = 100
|
| 95 |
+
self.seq_progress: dict = {}
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def _ctx(units):
|
| 99 |
+
return WinContext(signals=_FakeSignals(), render_state={"units_summary": units})
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
def test_waypoint_sequence_advances_only_in_order():
|
| 103 |
+
sig = _FakeSignals()
|
| 104 |
+
pts = [{"x": 110, "y": 8}, {"x": 110, "y": 32}]
|
| 105 |
+
spec = {"waypoint_sequence": {"id": "t", "n": 2, "radius": 7, "points": pts}}
|
| 106 |
+
|
| 107 |
+
# Visit W2 first — must NOT satisfy
|
| 108 |
+
ctx = WinContext(
|
| 109 |
+
signals=sig,
|
| 110 |
+
render_state={"units_summary": [
|
| 111 |
+
{"cell_x": 110, "cell_y": 32}, {"cell_x": 111, "cell_y": 32},
|
| 112 |
+
]},
|
| 113 |
+
)
|
| 114 |
+
assert evaluate(spec, ctx) is False
|
| 115 |
+
# idx must still be 0 (no premature advance from W2-only presence)
|
| 116 |
+
assert sig.seq_progress.get("t", 0) == 0
|
| 117 |
+
|
| 118 |
+
# Now visit W1
|
| 119 |
+
ctx = WinContext(
|
| 120 |
+
signals=sig,
|
| 121 |
+
render_state={"units_summary": [
|
| 122 |
+
{"cell_x": 110, "cell_y": 8}, {"cell_x": 111, "cell_y": 8},
|
| 123 |
+
]},
|
| 124 |
+
)
|
| 125 |
+
assert evaluate(spec, ctx) is False # W1 advances; W2 not yet
|
| 126 |
+
assert sig.seq_progress["t"] == 1
|
| 127 |
+
|
| 128 |
+
# Then visit W2 — sequence completes
|
| 129 |
+
ctx = WinContext(
|
| 130 |
+
signals=sig,
|
| 131 |
+
render_state={"units_summary": [
|
| 132 |
+
{"cell_x": 110, "cell_y": 32}, {"cell_x": 111, "cell_y": 32},
|
| 133 |
+
]},
|
| 134 |
+
)
|
| 135 |
+
assert evaluate(spec, ctx) is True
|
| 136 |
+
assert sig.seq_progress["t"] == 2
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def test_waypoint_sequence_requires_min_n_units():
|
| 140 |
+
sig = _FakeSignals()
|
| 141 |
+
pts = [{"x": 110, "y": 8}, {"x": 110, "y": 32}]
|
| 142 |
+
spec = {"waypoint_sequence": {"id": "t2", "n": 2, "radius": 7, "points": pts}}
|
| 143 |
+
|
| 144 |
+
# Only 1 unit at W1 — does not advance
|
| 145 |
+
ctx = WinContext(
|
| 146 |
+
signals=sig,
|
| 147 |
+
render_state={"units_summary": [{"cell_x": 110, "cell_y": 8}]},
|
| 148 |
+
)
|
| 149 |
+
assert evaluate(spec, ctx) is False
|
| 150 |
+
assert sig.seq_progress.get("t2", 0) == 0
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 154 |
+
def test_pack_runs_and_donothing_loses(level):
|
| 155 |
+
from openra_bench.eval_core import run_level
|
| 156 |
+
|
| 157 |
+
c = compile_level(load_pack(PACK), level)
|
| 158 |
+
assert c.map_supported
|
| 159 |
+
res = run_level(c, lambda rs, C: [C.observe()], seed=1)
|
| 160 |
+
assert res.outcome == "loss", (
|
| 161 |
+
f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
|
| 162 |
+
)
|
|
@@ -82,6 +82,22 @@ NOT_APPLICABLE = {
|
|
| 82 |
"instead of handed coordinates). Adding seed-driven spawn variation "
|
| 83 |
"would introduce a second uncontrolled variable and break the clean "
|
| 84 |
"medium→hard attribution.",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
}
|
| 86 |
|
| 87 |
# No-adversary maps: spawn variation applies but a force-loss
|
|
|
|
| 82 |
"instead of handed coordinates). Adding seed-driven spawn variation "
|
| 83 |
"would introduce a second uncontrolled variable and break the clean "
|
| 84 |
"medium→hard attribution.",
|
| 85 |
+
# PR #8 from yiyu-tian: 4 packs that tighten one decision axis per
|
| 86 |
+
# tier rather than per-spawn variation. UPGRADED would force a
|
| 87 |
+
# second variable per tier and break the clean attribution model.
|
| 88 |
+
"perception-count-the-threat-small-k": "hard tightens K (2→3→4) "
|
| 89 |
+
"and the positional non-obviousness of each squad — adding spawn "
|
| 90 |
+
"variation would dilute the count-and-infer signal. Companion to "
|
| 91 |
+
"perception-count-the-threat (UPGRADED above), which IS spawn-varied.",
|
| 92 |
+
"coordination-ordered-rendezvous": "hard tightens the waypoint "
|
| 93 |
+
"count (2→3→4) and the overall deadline; spawn variation would "
|
| 94 |
+
"compete with the order-enforcement axis for attribution.",
|
| 95 |
+
"tempo-strike-window": "hard tightens the lull length, follow-up "
|
| 96 |
+
"window, and attrition cap (cat-c11 canonical tempo design). "
|
| 97 |
+
"Spawn variation would dilute the tempo-discipline signal.",
|
| 98 |
+
"risk-blockade-bypass": "hard adds a tempo gate (after_ticks) and "
|
| 99 |
+
"shrinks the attrition cap; spawn variation would muddy the "
|
| 100 |
+
"corridor-vs-detour route-choice signal.",
|
| 101 |
}
|
| 102 |
|
| 103 |
# No-adversary maps: spawn variation applies but a force-loss
|
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""perception-count-the-threat: exact-K enemy discovery under fog.
|
| 2 |
+
|
| 3 |
+
Structural tests assert the predicate shape — each level discriminates
|
| 4 |
+
by the exact K (enemies_discovered_gte) and has a real fail_condition.
|
| 5 |
+
Run-smoke test confirms the pack loads and a do-nothing agent loses on
|
| 6 |
+
the timeout (proves fail_condition is wired through compile()).
|
| 7 |
+
|
| 8 |
+
Module-level importorskip because openra_bench.scenarios imports the
|
| 9 |
+
Rust adapter at import time (schema.py:15), so even pure-Python
|
| 10 |
+
structural tests need the wheel to load the package. Same pattern as
|
| 11 |
+
test_building_planning.py.
|
| 12 |
+
"""
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
|
| 17 |
+
import pytest
|
| 18 |
+
|
| 19 |
+
pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
|
| 20 |
+
|
| 21 |
+
from openra_bench.scenarios import load_pack
|
| 22 |
+
from openra_bench.scenarios.loader import compile_level
|
| 23 |
+
|
| 24 |
+
PACK = (
|
| 25 |
+
Path(__file__).parent.parent
|
| 26 |
+
/ "openra_bench"
|
| 27 |
+
/ "scenarios"
|
| 28 |
+
/ "packs"
|
| 29 |
+
/ "perception-count-the-threat-small-k.yaml"
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
EXPECTED_K = {"easy": 2, "medium": 3, "hard": 4}
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _win_clauses(c):
|
| 36 |
+
return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def _fail_clauses(c):
|
| 40 |
+
return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 44 |
+
def test_level_predicates_isolate_exact_count(level):
|
| 45 |
+
pack = load_pack(PACK)
|
| 46 |
+
assert pack.meta.capability == "perception"
|
| 47 |
+
c = compile_level(pack, level)
|
| 48 |
+
|
| 49 |
+
win = _win_clauses(c)
|
| 50 |
+
# exact K target for this level
|
| 51 |
+
ks = [cl["enemies_discovered_gte"] for cl in win if "enemies_discovered_gte" in cl]
|
| 52 |
+
assert ks == [EXPECTED_K[level]], (
|
| 53 |
+
f"{level}: expected enemies_discovered_gte={EXPECTED_K[level]}, got {ks}"
|
| 54 |
+
)
|
| 55 |
+
# tight clock binds (< max_turns * 90)
|
| 56 |
+
wt = [cl["within_ticks"] for cl in win if "within_ticks" in cl][0]
|
| 57 |
+
assert wt < c.max_turns * 90, f"{level}: within_ticks {wt} doesn't bind"
|
| 58 |
+
# attrition cap present
|
| 59 |
+
assert any("units_lost_lte" in cl for cl in win), (
|
| 60 |
+
f"{level}: missing units_lost_lte attrition cap"
|
| 61 |
+
)
|
| 62 |
+
|
| 63 |
+
# every level can LOSE (timeout-fail at min)
|
| 64 |
+
fail = _fail_clauses(c)
|
| 65 |
+
assert any("after_ticks" in cl for cl in fail), (
|
| 66 |
+
f"{level}: missing timeout in fail_condition"
|
| 67 |
+
)
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def test_k_increases_monotonically_with_difficulty():
|
| 71 |
+
pack = load_pack(PACK)
|
| 72 |
+
ks = []
|
| 73 |
+
for level in ("easy", "medium", "hard"):
|
| 74 |
+
c = compile_level(pack, level)
|
| 75 |
+
ks.append([cl["enemies_discovered_gte"] for cl in _win_clauses(c) if "enemies_discovered_gte" in cl][0])
|
| 76 |
+
assert ks == sorted(ks) and ks[0] < ks[-1], (
|
| 77 |
+
f"K must scale with difficulty; got {ks}"
|
| 78 |
+
)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def test_hidden_actor_count_matches_K():
|
| 82 |
+
"""K hidden enemies must actually exist on the map — otherwise the
|
| 83 |
+
win is unsatisfiable."""
|
| 84 |
+
pack = load_pack(PACK)
|
| 85 |
+
for level in ("easy", "medium", "hard"):
|
| 86 |
+
c = compile_level(pack, level)
|
| 87 |
+
enemy_actors = [a for a in c.scenario.actors if a.owner == "enemy"]
|
| 88 |
+
# count distinct enemy *positions* (each group is a discoverable squad)
|
| 89 |
+
positions = {(a.position[0], a.position[1]) for a in enemy_actors}
|
| 90 |
+
assert len(positions) >= EXPECTED_K[level], (
|
| 91 |
+
f"{level}: only {len(positions)} enemy positions on map, "
|
| 92 |
+
f"win needs {EXPECTED_K[level]}"
|
| 93 |
+
)
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 97 |
+
def test_pack_runs_and_donothing_loses(level):
|
| 98 |
+
from openra_bench.eval_core import run_level
|
| 99 |
+
|
| 100 |
+
c = compile_level(load_pack(PACK), level)
|
| 101 |
+
assert c.map_supported
|
| 102 |
+
res = run_level(c, lambda rs, C: [C.observe()], seed=1)
|
| 103 |
+
# do-nothing must lose on the timeout (proves fail_condition is live)
|
| 104 |
+
assert res.outcome == "loss", (
|
| 105 |
+
f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
|
| 106 |
+
)
|
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""risk-blockade-bypass: route-risk decision with two visible routes
|
| 2 |
+
(heavy corridor vs longer detour).
|
| 3 |
+
|
| 4 |
+
Structural tests verify the win predicate combines a positional
|
| 5 |
+
objective (reach_region) with a binding attrition cap and clock —
|
| 6 |
+
the three pieces that make this a risk decision, not a navigation
|
| 7 |
+
or combat test. Hard adds a tempo gate (after_ticks) tightening the
|
| 8 |
+
risk calculus (can't speedrun).
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import pytest
|
| 15 |
+
|
| 16 |
+
# openra_bench.scenarios eagerly imports the Rust adapter at module
|
| 17 |
+
# load (schema.py:15), so collection fails without the wheel. Skip the
|
| 18 |
+
# whole module if the env is missing — matches test_building_planning.
|
| 19 |
+
pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
|
| 20 |
+
|
| 21 |
+
from openra_bench.scenarios import load_pack
|
| 22 |
+
from openra_bench.scenarios.loader import compile_level
|
| 23 |
+
|
| 24 |
+
PACK = (
|
| 25 |
+
Path(__file__).parent.parent
|
| 26 |
+
/ "openra_bench"
|
| 27 |
+
/ "scenarios"
|
| 28 |
+
/ "packs"
|
| 29 |
+
/ "risk-blockade-bypass.yaml"
|
| 30 |
+
)
|
| 31 |
+
|
| 32 |
+
# (deadline, loss_cap, after_ticks_gate_or_None)
|
| 33 |
+
# Tempo gate dropped from hard to keep the difficulty axis on route-
|
| 34 |
+
# risk only (CLAUDE.md C.10: one new controlled variable per tier).
|
| 35 |
+
EXPECTED = {
|
| 36 |
+
"easy": (3500, 4, None),
|
| 37 |
+
"medium": (2800, 2, None),
|
| 38 |
+
"hard": (2400, 1, None),
|
| 39 |
+
}
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _win_clauses(c):
|
| 43 |
+
return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _fail_clauses(c):
|
| 47 |
+
return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 51 |
+
def test_level_encodes_reach_with_attrition_and_clock(level):
|
| 52 |
+
pack = load_pack(PACK)
|
| 53 |
+
assert pack.meta.capability == "reasoning"
|
| 54 |
+
c = compile_level(pack, level)
|
| 55 |
+
|
| 56 |
+
win = _win_clauses(c)
|
| 57 |
+
reach = [cl["reach_region"] for cl in win if "reach_region" in cl]
|
| 58 |
+
deadline = [cl["within_ticks"] for cl in win if "within_ticks" in cl]
|
| 59 |
+
loss_cap = [cl["units_lost_lte"] for cl in win if "units_lost_lte" in cl]
|
| 60 |
+
after = [cl["after_ticks"] for cl in win if "after_ticks" in cl]
|
| 61 |
+
|
| 62 |
+
assert len(reach) == 1, f"{level}: must reach exactly one region"
|
| 63 |
+
assert reach[0]["x"] == 110, f"{level}: objective should be at the east marker"
|
| 64 |
+
assert len(deadline) == 1 and len(loss_cap) == 1, (
|
| 65 |
+
f"{level}: must have both within_ticks and units_lost_lte"
|
| 66 |
+
)
|
| 67 |
+
|
| 68 |
+
exp_deadline, exp_loss, exp_after = EXPECTED[level]
|
| 69 |
+
assert deadline[0] == exp_deadline
|
| 70 |
+
assert loss_cap[0] == exp_loss
|
| 71 |
+
if exp_after is None:
|
| 72 |
+
assert after == [], f"{level}: should not have an after_ticks gate"
|
| 73 |
+
else:
|
| 74 |
+
assert after == [exp_after], f"{level}: expected after_ticks={exp_after}, got {after}"
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def test_attrition_cap_tightens_with_difficulty():
|
| 78 |
+
caps = [EXPECTED[lv][1] for lv in ("easy", "medium", "hard")]
|
| 79 |
+
assert caps == [4, 2, 1] and caps == sorted(caps, reverse=True), (
|
| 80 |
+
f"attrition cap must tighten 4->2->1: {caps}"
|
| 81 |
+
)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def test_deadline_tightens_with_difficulty():
|
| 85 |
+
deadlines = [EXPECTED[lv][0] for lv in ("easy", "medium", "hard")]
|
| 86 |
+
assert deadlines == sorted(deadlines, reverse=True), (
|
| 87 |
+
f"deadline must tighten: {deadlines}"
|
| 88 |
+
)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def test_both_routes_visibly_present_on_map():
|
| 92 |
+
"""The map must contain BOTH the heavy-corridor garrison (mid-east)
|
| 93 |
+
and the light-detour picket (north) — otherwise it isn't a route
|
| 94 |
+
choice scenario."""
|
| 95 |
+
pack = load_pack(PACK)
|
| 96 |
+
for level in ("easy", "medium", "hard"):
|
| 97 |
+
c = compile_level(pack, level)
|
| 98 |
+
enemy = [a for a in c.scenario.actors if a.owner == "enemy"]
|
| 99 |
+
# heavy corridor garrison at y~20 (mid-latitude)
|
| 100 |
+
corridor = [a for a in enemy if 15 <= a.position[1] <= 25 and 40 <= a.position[0] <= 70]
|
| 101 |
+
# light detour picket at y~5 (top)
|
| 102 |
+
detour = [a for a in enemy if a.position[1] <= 10 and 40 <= a.position[0] <= 70]
|
| 103 |
+
assert corridor, f"{level}: missing heavy corridor garrison"
|
| 104 |
+
assert detour, f"{level}: missing light detour picket"
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 108 |
+
def test_pack_runs_and_donothing_loses(level):
|
| 109 |
+
from openra_bench.eval_core import run_level
|
| 110 |
+
|
| 111 |
+
c = compile_level(load_pack(PACK), level)
|
| 112 |
+
assert c.map_supported
|
| 113 |
+
res = run_level(c, lambda rs, C: [C.observe()], seed=1)
|
| 114 |
+
assert res.outcome == "loss", (
|
| 115 |
+
f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
|
| 116 |
+
)
|
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""tempo-strike-window: forced lull then kill quota.
|
| 2 |
+
|
| 3 |
+
Structural tests verify every level encodes (a) an after_ticks lull
|
| 4 |
+
gate, (b) a kill quota, (c) a within_ticks deadline, (d) an attrition
|
| 5 |
+
cap, AND (e) a "premature engagement" fail clause keyed off
|
| 6 |
+
units_killed_gte inside the lull window.
|
| 7 |
+
"""
|
| 8 |
+
from __future__ import annotations
|
| 9 |
+
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
|
| 12 |
+
import pytest
|
| 13 |
+
|
| 14 |
+
# openra_bench.scenarios eagerly imports the Rust adapter at module
|
| 15 |
+
# load (schema.py:15), so collection fails without the wheel. Skip the
|
| 16 |
+
# whole module if the env is missing — matches test_building_planning.
|
| 17 |
+
pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
|
| 18 |
+
|
| 19 |
+
from openra_bench.scenarios import load_pack
|
| 20 |
+
from openra_bench.scenarios.loader import compile_level
|
| 21 |
+
|
| 22 |
+
PACK = (
|
| 23 |
+
Path(__file__).parent.parent
|
| 24 |
+
/ "openra_bench"
|
| 25 |
+
/ "scenarios"
|
| 26 |
+
/ "packs"
|
| 27 |
+
/ "tempo-strike-window.yaml"
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
# (lull_until, kill_target, deadline, loss_cap)
|
| 31 |
+
EXPECTED = {
|
| 32 |
+
"easy": (2000, 4, 4200, 3),
|
| 33 |
+
"medium": (1800, 5, 3600, 2),
|
| 34 |
+
"hard": (1500, 6, 3000, 1),
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _win_clauses(c):
|
| 39 |
+
return dict(c.win_condition.__pydantic_extra__ or {})["all_of"]
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _fail_clauses(c):
|
| 43 |
+
return dict(c.fail_condition.__pydantic_extra__ or {})["any_of"]
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 47 |
+
def test_level_encodes_lull_kill_deadline_attrition(level):
|
| 48 |
+
pack = load_pack(PACK)
|
| 49 |
+
assert pack.meta.capability == "reasoning"
|
| 50 |
+
c = compile_level(pack, level)
|
| 51 |
+
|
| 52 |
+
win = _win_clauses(c)
|
| 53 |
+
after = [cl["after_ticks"] for cl in win if "after_ticks" in cl]
|
| 54 |
+
killed = [cl["units_killed_gte"] for cl in win if "units_killed_gte" in cl]
|
| 55 |
+
within = [cl["within_ticks"] for cl in win if "within_ticks" in cl]
|
| 56 |
+
lost = [cl["units_lost_lte"] for cl in win if "units_lost_lte" in cl]
|
| 57 |
+
|
| 58 |
+
assert len(after) == 1 and len(killed) == 1 and len(within) == 1 and len(lost) == 1, (
|
| 59 |
+
f"{level}: must have exactly one of each (after_ticks/units_killed_gte/"
|
| 60 |
+
f"within_ticks/units_lost_lte)"
|
| 61 |
+
)
|
| 62 |
+
lull_until, kill_target, deadline, loss_cap = EXPECTED[level]
|
| 63 |
+
assert after[0] == lull_until, f"{level}: lull_until expected {lull_until}, got {after[0]}"
|
| 64 |
+
assert killed[0] == kill_target, f"{level}: kill_target expected {kill_target}, got {killed[0]}"
|
| 65 |
+
assert within[0] == deadline, f"{level}: deadline expected {deadline}, got {within[0]}"
|
| 66 |
+
assert lost[0] == loss_cap, f"{level}: loss_cap expected {loss_cap}, got {lost[0]}"
|
| 67 |
+
assert after[0] < within[0], f"{level}: lull must end before deadline"
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 71 |
+
def test_fail_condition_includes_premature_engagement(level):
|
| 72 |
+
"""The 'premature engagement' fail = units_killed_gte:N AND
|
| 73 |
+
within_ticks:lull-1 — i.e. having too many kills before the lull
|
| 74 |
+
elapses immediately loses."""
|
| 75 |
+
pack = load_pack(PACK)
|
| 76 |
+
c = compile_level(pack, level)
|
| 77 |
+
fail = _fail_clauses(c)
|
| 78 |
+
lull_until = EXPECTED[level][0]
|
| 79 |
+
|
| 80 |
+
has_premature = False
|
| 81 |
+
for cl in fail:
|
| 82 |
+
if "all_of" in cl:
|
| 83 |
+
inner = cl["all_of"]
|
| 84 |
+
has_killed = any("units_killed_gte" in x for x in inner)
|
| 85 |
+
within_cap = [x["within_ticks"] for x in inner if "within_ticks" in x]
|
| 86 |
+
if has_killed and within_cap and within_cap[0] < lull_until:
|
| 87 |
+
has_premature = True
|
| 88 |
+
break
|
| 89 |
+
assert has_premature, (
|
| 90 |
+
f"{level}: missing premature-engagement fail "
|
| 91 |
+
f"(units_killed_gte AND within_ticks < {lull_until})"
|
| 92 |
+
)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def test_difficulty_tightens_lull_kill_deadline_attrition():
|
| 96 |
+
"""easy -> medium -> hard: shorter lull, higher kill bar, shorter
|
| 97 |
+
deadline, tighter attrition cap."""
|
| 98 |
+
lulls = [EXPECTED[lv][0] for lv in ("easy", "medium", "hard")]
|
| 99 |
+
kills = [EXPECTED[lv][1] for lv in ("easy", "medium", "hard")]
|
| 100 |
+
deadlines = [EXPECTED[lv][2] for lv in ("easy", "medium", "hard")]
|
| 101 |
+
losses = [EXPECTED[lv][3] for lv in ("easy", "medium", "hard")]
|
| 102 |
+
|
| 103 |
+
assert lulls == sorted(lulls, reverse=True), f"lull should shrink: {lulls}"
|
| 104 |
+
assert kills == sorted(kills), f"kill target should grow: {kills}"
|
| 105 |
+
assert deadlines == sorted(deadlines, reverse=True), f"deadline should shrink: {deadlines}"
|
| 106 |
+
assert losses == sorted(losses, reverse=True), f"loss cap should shrink: {losses}"
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
@pytest.mark.parametrize("level", ["easy", "medium", "hard"])
|
| 110 |
+
def test_pack_runs_and_donothing_loses(level):
|
| 111 |
+
from openra_bench.eval_core import run_level
|
| 112 |
+
|
| 113 |
+
c = compile_level(load_pack(PACK), level)
|
| 114 |
+
assert c.map_supported
|
| 115 |
+
res = run_level(c, lambda rs, C: [C.observe()], seed=1)
|
| 116 |
+
assert res.outcome == "loss", (
|
| 117 |
+
f"{level}: do-nothing should LOSE on timeout, got {res.outcome}"
|
| 118 |
+
)
|