Spaces:
Running
Running
feat(scenario): scout-deny-enemy-vision — counter-recon, kill enemy scouts before they report (military counter-reconnaissance)
Browse filesREASONING pack: enemy scouts hold observation posts watching the
agent's base; left alone they complete their report. The agent's
HoldFire strike force must intercept and kill every scout before the
report window (within_ticks) closes. Stall and wrong-path LOSE
(0 kills); committing the whole force to one recon vector LOSES on
medium/hard (the other scout reports). Only splitting to cover every
vector WINS. Hard defines two seed-driven spawn_point base groups.
openra_bench/scenarios/packs/scout-deny-enemy-vision.yaml
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# scout-deny-enemy-vision — REASONING, counter-reconnaissance:
|
| 2 |
+
# intercept and kill the enemy scout(s) before they report
|
| 3 |
+
# (Wave-12).
|
| 4 |
+
#
|
| 5 |
+
# The capability under test is COUNTER-RECON — denying the enemy
|
| 6 |
+
# vision of your base. An enemy scout has taken up an observation
|
| 7 |
+
# post overlooking the agent's base; left alone it completes its
|
| 8 |
+
# report (it "survives N ticks in sight"). The agent must locate and
|
| 9 |
+
# DESTROY every enemy scout before the report window closes. Sitting
|
| 10 |
+
# still lets the report complete; covering only ONE recon vector
|
| 11 |
+
# leaves the other scout's report intact.
|
| 12 |
+
#
|
| 13 |
+
# Real-world anchors:
|
| 14 |
+
# • Military counter-reconnaissance: the counter-recon screen
|
| 15 |
+
# hunts and destroys the enemy's scouts/observation posts before
|
| 16 |
+
# they can report the main body's disposition.
|
| 17 |
+
# • SC2 scout denial: killing the opponent's scouting worker /
|
| 18 |
+
# overlord before it sees your build — deny the information.
|
| 19 |
+
# • Intrusion prevention: detect and TERMINATE the reconnaissance
|
| 20 |
+
# probe before it exfiltrates a map of the defended network;
|
| 21 |
+
# a passive posture that merely watches the probe loses the
|
| 22 |
+
# information war.
|
| 23 |
+
#
|
| 24 |
+
# Mechanism (the load-bearing design):
|
| 25 |
+
# • The enemy scouts are `e3` actors at `stance:0` (HoldFire) —
|
| 26 |
+
# they hold an observation post and do NOT fight or advance, so
|
| 27 |
+
# the agent must DRIVE every engagement (a stall cannot bait a
|
| 28 |
+
# free kill, and the scouts cannot self-deliver into the
|
| 29 |
+
# agent's force).
|
| 30 |
+
# • The agent's strike force is `stance:0` (HoldFire) too — it
|
| 31 |
+
# never auto-hunts the map, so a stall / observe-only policy
|
| 32 |
+
# can NOT clear the scouts for free; the agent must issue the
|
| 33 |
+
# explicit attack_move + attack_unit interception orders.
|
| 34 |
+
# • The win is `units_killed_gte:N` (kill EVERY scout) AND
|
| 35 |
+
# `within_ticks:K` — K is the report window: a scout still
|
| 36 |
+
# alive at K has "survived N ticks in sight" and completed its
|
| 37 |
+
# report ⇒ the agent loses.
|
| 38 |
+
# • On medium/hard there are TWO scouts on TWO separate recon
|
| 39 |
+
# vectors (a north lane and a south lane). Committing the whole
|
| 40 |
+
# force to ONE lane kills only one scout — the other completes
|
| 41 |
+
# its report ⇒ the kill bar (N=2) is unmet ⇒ LOSS. The
|
| 42 |
+
# counter-recon skill is to cover EVERY vector.
|
| 43 |
+
#
|
| 44 |
+
# Discrimination (four-policy bar — scripted, no model needed):
|
| 45 |
+
# • stall (observe-only): the HoldFire strike force never engages;
|
| 46 |
+
# 0 kills; every scout survives to K ⇒ the `after_ticks`
|
| 47 |
+
# deadline bites as a real reachable timeout LOSS.
|
| 48 |
+
# • one-lane (commit the whole force to a single recon vector):
|
| 49 |
+
# kills only the scout on that lane; the other scout completes
|
| 50 |
+
# its report ⇒ the kill bar is unmet ⇒ LOSS (medium/hard).
|
| 51 |
+
# • wrong-path (sweep the wrong lane / chase the far enemy
|
| 52 |
+
# marker): never reaches a real scout ⇒ 0 (or too few) kills
|
| 53 |
+
# ⇒ LOSS.
|
| 54 |
+
# • intended counter-recon (split the force to cover EVERY recon
|
| 55 |
+
# vector; attack_move to each scout's post and attack_unit it
|
| 56 |
+
# to death): every scout killed before K ⇒ WIN.
|
| 57 |
+
#
|
| 58 |
+
# Engine facts used (CLAUDE.md / prompt_v2 codex / verified against
|
| 59 |
+
# the installed openra_train wheel, 2026-05-20):
|
| 60 |
+
# • `e3` rocket soldiers surface reliably in `enemy_positions` /
|
| 61 |
+
# `enemy_summary` once in vision (CLAUDE.md notes some `e1`
|
| 62 |
+
# cells do not surface — `e3` does).
|
| 63 |
+
# • `stance:0` HoldFire — a unit never auto-engages, even when
|
| 64 |
+
# attacked. The scouts hold their post; the agent strike force
|
| 65 |
+
# does not self-deliver. Explicit `attack_unit` / `attack_move`
|
| 66 |
+
# orders override stance (CLAUDE.md stance semantics).
|
| 67 |
+
# • Tick alignment (~90 ticks/turn): within_ticks ≤ 93+90·
|
| 68 |
+
# (max_turns−1); fail after_ticks = within_ticks+1 ⇒ a
|
| 69 |
+
# non-finisher is a real reachable LOSS, never a DRAW.
|
| 70 |
+
# • A far inert enemy `fact` marker keeps the engine from
|
| 71 |
+
# auto-`done`ing on enemy-elimination before the win/fail
|
| 72 |
+
# evaluator runs (CLAUDE.md auto-done footgun).
|
| 73 |
+
# • Hard tier: two agent spawn_point groups (NORTH base y=10..14
|
| 74 |
+
# / SOUTH base y=26..30); the agent's strike force is
|
| 75 |
+
# duplicated across both groups at the matching latitude. The
|
| 76 |
+
# two scouts (no spawn_point) place every seed, so a memorised
|
| 77 |
+
# "sweep these two cells from (20,20)" opening cannot
|
| 78 |
+
# generalise — the interception geometry flips per seed.
|
| 79 |
+
#
|
| 80 |
+
# Validate (no model / no network):
|
| 81 |
+
# python3 -m pytest tests/test_scout_deny_enemy_vision.py -q
|
| 82 |
+
|
| 83 |
+
meta:
|
| 84 |
+
id: scout-deny-enemy-vision
|
| 85 |
+
title: 'Counter-Recon — Deny Enemy Vision: Kill the Scouts Before They Report'
|
| 86 |
+
capability: reasoning
|
| 87 |
+
real_world_meaning: >
|
| 88 |
+
An enemy scout has taken up an observation post overlooking your
|
| 89 |
+
base; left alone it completes its report and the enemy gains a
|
| 90 |
+
full picture of your disposition. The agent must locate and
|
| 91 |
+
DESTROY every enemy scout before the report window closes.
|
| 92 |
+
Sitting still lets the report complete; covering only ONE recon
|
| 93 |
+
vector leaves the other scout's report intact. The pack tests
|
| 94 |
+
counter-reconnaissance: actively denying the adversary vision by
|
| 95 |
+
hunting down EVERY scout, not merely defending one approach.
|
| 96 |
+
robotics_analogue: >
|
| 97 |
+
Counter-surveillance / intrusion prevention: an adversary's
|
| 98 |
+
reconnaissance probes are mapping the defended site, and each
|
| 99 |
+
probe exfiltrates its findings once it has observed long enough.
|
| 100 |
+
The defender must detect and neutralise EVERY probe before its
|
| 101 |
+
report window elapses — a passive posture that merely watches
|
| 102 |
+
the probes loses the information war, and neutralising one probe
|
| 103 |
+
while ignoring another still leaks the map.
|
| 104 |
+
benchmark_anchor:
|
| 105 |
+
- "military counter-reconnaissance"
|
| 106 |
+
- "SC2 scout denial"
|
| 107 |
+
- "intrusion prevention"
|
| 108 |
+
author: openra-bench
|
| 109 |
+
|
| 110 |
+
base_map: rush-hour-arena
|
| 111 |
+
|
| 112 |
+
base:
|
| 113 |
+
agent: {faction: allies, cash: 0}
|
| 114 |
+
enemy: {faction: soviet, cash: 0}
|
| 115 |
+
tools: [observe, move_units, attack_unit, attack_move, stop]
|
| 116 |
+
planning: true
|
| 117 |
+
termination: {max_ticks: 9000}
|
| 118 |
+
actors: []
|
| 119 |
+
|
| 120 |
+
levels:
|
| 121 |
+
# ── EASY ────────────────────────────────────────────────────────
|
| 122 |
+
# The bare counter-recon skill: ONE enemy scout on an observation
|
| 123 |
+
# post forward of the base. The agent's HoldFire strike force must
|
| 124 |
+
# drive out and kill it before the report window (within_ticks)
|
| 125 |
+
# closes. Stall LOSES (HoldFire force never engages; the scout
|
| 126 |
+
# survives to K). A decisive intercept WINS.
|
| 127 |
+
easy:
|
| 128 |
+
description: >
|
| 129 |
+
An enemy scout (rocket soldier) holds an observation post
|
| 130 |
+
forward of your base at (55, 20), watching your disposition.
|
| 131 |
+
Your strike force (2 medium tanks) holds fire — it will not
|
| 132 |
+
engage on its own. Drive the force out and KILL the scout
|
| 133 |
+
before tick 1100 (the report window). If the scout is still
|
| 134 |
+
alive at the deadline it has completed its report and you
|
| 135 |
+
LOSE. Stalling (observe only) LOSES — the scout survives the
|
| 136 |
+
window. Sweeping the wrong direction LOSES — you never reach
|
| 137 |
+
the scout in time.
|
| 138 |
+
overrides:
|
| 139 |
+
actors:
|
| 140 |
+
# Agent strike force — HoldFire so it never auto-hunts; the
|
| 141 |
+
# agent must issue the interception orders.
|
| 142 |
+
- {type: 2tnk, owner: agent, position: [20, 19], stance: 0}
|
| 143 |
+
- {type: 2tnk, owner: agent, position: [20, 21], stance: 0}
|
| 144 |
+
# Enemy scout — e3 on a HoldFire observation post.
|
| 145 |
+
- {type: e3, owner: enemy, position: [55, 20], stance: 0}
|
| 146 |
+
# Far inert enemy marker — anti-DRAW.
|
| 147 |
+
- {type: fact, owner: enemy, position: [124, 20]}
|
| 148 |
+
win_condition:
|
| 149 |
+
all_of:
|
| 150 |
+
- {units_killed_gte: 1}
|
| 151 |
+
- {within_ticks: 1100}
|
| 152 |
+
fail_condition:
|
| 153 |
+
any_of:
|
| 154 |
+
- {after_ticks: 1101}
|
| 155 |
+
max_turns: 25
|
| 156 |
+
|
| 157 |
+
# ── MEDIUM ──────────────────────────────────────────────────────
|
| 158 |
+
# +1 controlled variable: TWO enemy scouts on TWO separate recon
|
| 159 |
+
# vectors (a north lane y=12 and a south lane y=28). The kill bar
|
| 160 |
+
# rises to 2 — the agent must cover BOTH vectors. Committing the
|
| 161 |
+
# whole force to one lane kills one scout; the other completes its
|
| 162 |
+
# report ⇒ LOSS. The strike force is sized so a SPLIT (one
|
| 163 |
+
# detachment per lane) clears both scouts inside the window.
|
| 164 |
+
medium:
|
| 165 |
+
description: >
|
| 166 |
+
TWO enemy scouts (rocket soldiers) hold observation posts on
|
| 167 |
+
two separate recon vectors — one on the NORTH lane at
|
| 168 |
+
(55, 12) and one on the SOUTH lane at (55, 28). Your strike
|
| 169 |
+
force (4 medium tanks) holds fire. SPLIT the force to cover
|
| 170 |
+
BOTH vectors and KILL BOTH scouts before tick 1300. Any
|
| 171 |
+
scout still alive at the deadline has completed its report
|
| 172 |
+
and you LOSE. Committing the whole force to one lane kills
|
| 173 |
+
only one scout — the other reports ⇒ LOSS. Stalling LOSES.
|
| 174 |
+
overrides:
|
| 175 |
+
actors:
|
| 176 |
+
# Agent strike force — two detachments, one per lane.
|
| 177 |
+
- {type: 2tnk, owner: agent, position: [20, 11], stance: 0}
|
| 178 |
+
- {type: 2tnk, owner: agent, position: [20, 13], stance: 0}
|
| 179 |
+
- {type: 2tnk, owner: agent, position: [20, 27], stance: 0}
|
| 180 |
+
- {type: 2tnk, owner: agent, position: [20, 29], stance: 0}
|
| 181 |
+
# Two enemy scouts on two recon vectors.
|
| 182 |
+
- {type: e3, owner: enemy, position: [55, 12], stance: 0}
|
| 183 |
+
- {type: e3, owner: enemy, position: [55, 28], stance: 0}
|
| 184 |
+
- {type: fact, owner: enemy, position: [124, 20]}
|
| 185 |
+
win_condition:
|
| 186 |
+
all_of:
|
| 187 |
+
- {units_killed_gte: 2}
|
| 188 |
+
- {within_ticks: 1300}
|
| 189 |
+
fail_condition:
|
| 190 |
+
any_of:
|
| 191 |
+
- {after_ticks: 1301}
|
| 192 |
+
max_turns: 25
|
| 193 |
+
|
| 194 |
+
# ── HARD ────────────────────────────────────────────────────────
|
| 195 |
+
# +2 controlled variables vs medium:
|
| 196 |
+
# 1. Tighter report window (~1100 ticks instead of ~1300) — the
|
| 197 |
+
# counter-recon response must be efficient; a dawdling split
|
| 198 |
+
# misses the clock.
|
| 199 |
+
# 2. TWO agent spawn_point groups (NORTH base y=10..14 / SOUTH
|
| 200 |
+
# base y=26..30) round-robined by seed. The two scouts
|
| 201 |
+
# (no spawn_point) place every seed at fixed posts, so the
|
| 202 |
+
# interception geometry — which detachment is nearer which
|
| 203 |
+
# scout — flips per seed; a memorised "north detachment
|
| 204 |
+
# takes the north scout" opening cannot generalise.
|
| 205 |
+
hard:
|
| 206 |
+
description: >
|
| 207 |
+
TWO enemy scouts (rocket soldiers) hold observation posts at
|
| 208 |
+
(55, 12) and (55, 28). Your strike force (4 medium tanks)
|
| 209 |
+
starts at ONE of two base latitudes (NORTH or SOUTH, chosen
|
| 210 |
+
by seed) and holds fire. SPLIT the force to cover BOTH recon
|
| 211 |
+
vectors and KILL BOTH scouts before tick 1100 — a tight
|
| 212 |
+
report window. Any scout still alive at the deadline has
|
| 213 |
+
completed its report and you LOSE. Committing the whole force
|
| 214 |
+
to one lane LOSES; stalling LOSES. The base latitude varies
|
| 215 |
+
by seed so a memorised interception plan cannot generalise.
|
| 216 |
+
overrides:
|
| 217 |
+
actors:
|
| 218 |
+
# spawn_point 0 — NORTH base (y=9..15)
|
| 219 |
+
- {type: 2tnk, owner: agent, position: [20, 9], stance: 0, spawn_point: 0}
|
| 220 |
+
- {type: 2tnk, owner: agent, position: [22, 11], stance: 0, spawn_point: 0}
|
| 221 |
+
- {type: 2tnk, owner: agent, position: [20, 13], stance: 0, spawn_point: 0}
|
| 222 |
+
- {type: 2tnk, owner: agent, position: [22, 15], stance: 0, spawn_point: 0}
|
| 223 |
+
# spawn_point 1 — SOUTH base (y=25..31)
|
| 224 |
+
- {type: 2tnk, owner: agent, position: [20, 25], stance: 0, spawn_point: 1}
|
| 225 |
+
- {type: 2tnk, owner: agent, position: [22, 27], stance: 0, spawn_point: 1}
|
| 226 |
+
- {type: 2tnk, owner: agent, position: [20, 29], stance: 0, spawn_point: 1}
|
| 227 |
+
- {type: 2tnk, owner: agent, position: [22, 31], stance: 0, spawn_point: 1}
|
| 228 |
+
# Two enemy scouts — fixed posts, place every seed.
|
| 229 |
+
- {type: e3, owner: enemy, position: [55, 12], stance: 0}
|
| 230 |
+
- {type: e3, owner: enemy, position: [55, 28], stance: 0}
|
| 231 |
+
- {type: fact, owner: enemy, position: [124, 20]}
|
| 232 |
+
win_condition:
|
| 233 |
+
all_of:
|
| 234 |
+
- {units_killed_gte: 2}
|
| 235 |
+
- {within_ticks: 1100}
|
| 236 |
+
fail_condition:
|
| 237 |
+
any_of:
|
| 238 |
+
- {after_ticks: 1101}
|
| 239 |
+
max_turns: 25
|
tests/test_hard_tier.py
CHANGED
|
@@ -1450,6 +1450,7 @@ UPGRADED = [
|
|
| 1450 |
"coord-cover-and-move", # hard: 2 agent spawn_point groups
|
| 1451 |
"combat-kite-and-pull", # hard: 2 agent spawn_point groups (Wave-12)
|
| 1452 |
"econ-tech-vs-expand-decision", # hard: 2 agent spawn_point groups (Wave-12)
|
|
|
|
| 1453 |
]
|
| 1454 |
|
| 1455 |
# Consciously NOT spawn-varied, with the reason (keeps the curation
|
|
|
|
| 1450 |
"coord-cover-and-move", # hard: 2 agent spawn_point groups
|
| 1451 |
"combat-kite-and-pull", # hard: 2 agent spawn_point groups (Wave-12)
|
| 1452 |
"econ-tech-vs-expand-decision", # hard: 2 agent spawn_point groups (Wave-12)
|
| 1453 |
+
"scout-deny-enemy-vision", # hard: 2 agent spawn_point groups (Wave-12)
|
| 1454 |
]
|
| 1455 |
|
| 1456 |
# Consciously NOT spawn-varied, with the reason (keeps the curation
|
tests/test_scout_deny_enemy_vision.py
ADDED
|
@@ -0,0 +1,325 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""scout-deny-enemy-vision — REASONING capability validation.
|
| 2 |
+
|
| 3 |
+
Counter-reconnaissance: enemy scouts hold observation posts watching
|
| 4 |
+
the agent's base; left alone they complete their report. The agent
|
| 5 |
+
must intercept and KILL every scout before the report window closes
|
| 6 |
+
(`within_ticks`). Sitting still lets the report complete; covering
|
| 7 |
+
only ONE recon vector leaves the other scout's report intact.
|
| 8 |
+
|
| 9 |
+
Bar (CLAUDE.md "no defect, no cheat, no draw"):
|
| 10 |
+
|
| 11 |
+
* stall (observe-only) LOSES every tier / every hard seed — the
|
| 12 |
+
HoldFire strike force never engages; 0 kills; the scouts
|
| 13 |
+
survive the window → the `after_ticks` deadline bites.
|
| 14 |
+
* wrong-path (sweep away from the scouts) LOSES every tier /
|
| 15 |
+
seed — no scout is reached → 0 kills.
|
| 16 |
+
* one-lane (commit the whole force to a single recon vector)
|
| 17 |
+
LOSES on medium/hard — kills one scout, the other completes
|
| 18 |
+
its report → the N=2 kill bar is unmet.
|
| 19 |
+
* intended counter-recon (split the force to cover EVERY recon
|
| 20 |
+
vector, attack_unit each scout) WINS every tier / every hard
|
| 21 |
+
seed.
|
| 22 |
+
* hard tier defines ≥2 agent spawn_point groups (NORTH / SOUTH
|
| 23 |
+
base) round-robined by seed so the interception geometry
|
| 24 |
+
cannot be memorised.
|
| 25 |
+
"""
|
| 26 |
+
|
| 27 |
+
from __future__ import annotations
|
| 28 |
+
|
| 29 |
+
from pathlib import Path
|
| 30 |
+
|
| 31 |
+
import pytest
|
| 32 |
+
|
| 33 |
+
pytest.importorskip("openra_train", reason="Rust env wheel not installed")
|
| 34 |
+
pytest.importorskip("openra_rl_training", reason="Rust env wheel not installed")
|
| 35 |
+
|
| 36 |
+
from openra_bench.eval_core import run_level
|
| 37 |
+
from openra_bench.scenarios import load_pack
|
| 38 |
+
from openra_bench.scenarios.loader import PACKS_DIR, compile_level
|
| 39 |
+
from openra_bench.scenarios.win_conditions import WinContext, evaluate
|
| 40 |
+
|
| 41 |
+
PACK = PACKS_DIR / "scout-deny-enemy-vision.yaml"
|
| 42 |
+
LEVELS = ("easy", "medium", "hard")
|
| 43 |
+
SEEDS = (1, 2, 3, 4)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
# ── scripted policies ───────────────────────────────────────────────
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _tanks(rs):
|
| 50 |
+
return [u for u in (rs.get("units_summary") or []) if u.get("type") == "2tnk"]
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _scouts(rs):
|
| 54 |
+
return [e for e in (rs.get("enemy_summary") or []) if (e.get("type") or "").lower() == "e3"]
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def _stall(rs, C):
|
| 58 |
+
"""Observe-only — the HoldFire force never engages → 0 kills →
|
| 59 |
+
the scouts survive the report window → LOSS."""
|
| 60 |
+
return [C.observe()]
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _wrong_path(rs, C):
|
| 64 |
+
"""Sweep toward the far enemy marker, away from the scouts. No
|
| 65 |
+
scout is ever reached → 0 kills → LOSS."""
|
| 66 |
+
own = _tanks(rs)
|
| 67 |
+
if not own:
|
| 68 |
+
return [C.observe()]
|
| 69 |
+
return [C.attack_move([str(u["id"]) for u in own], target_x=110, target_y=20)]
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def _one_lane(rs, C):
|
| 73 |
+
"""Commit the WHOLE force to a single recon vector (the north
|
| 74 |
+
scout). On medium/hard the south scout completes its report →
|
| 75 |
+
the N=2 kill bar is unmet → LOSS."""
|
| 76 |
+
own = _tanks(rs)
|
| 77 |
+
if not own:
|
| 78 |
+
return [C.observe()]
|
| 79 |
+
scouts = sorted(_scouts(rs), key=lambda e: e["cell_y"])
|
| 80 |
+
if scouts:
|
| 81 |
+
return [C.attack_unit([str(u["id"]) for u in own], str(scouts[0]["id"]))]
|
| 82 |
+
return [C.attack_move([str(u["id"]) for u in own], target_x=55, target_y=12)]
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _counter_recon(rs, C):
|
| 86 |
+
"""The intended capability — cover EVERY recon vector. With
|
| 87 |
+
vision, every tank attacks the nearest scout; without vision,
|
| 88 |
+
split the force (north half sweeps the north lane, south half
|
| 89 |
+
the south lane) so both observation posts are reached."""
|
| 90 |
+
own = _tanks(rs)
|
| 91 |
+
if not own:
|
| 92 |
+
return [C.observe()]
|
| 93 |
+
scouts = _scouts(rs)
|
| 94 |
+
cmds = []
|
| 95 |
+
if scouts:
|
| 96 |
+
for u in own:
|
| 97 |
+
s = min(
|
| 98 |
+
scouts,
|
| 99 |
+
key=lambda e: abs(e["cell_x"] - u["cell_x"])
|
| 100 |
+
+ abs(e["cell_y"] - u["cell_y"]),
|
| 101 |
+
)
|
| 102 |
+
cmds.append(C.attack_unit([str(u["id"])], str(s["id"])))
|
| 103 |
+
return cmds
|
| 104 |
+
sown = sorted(own, key=lambda u: u["cell_y"])
|
| 105 |
+
half = len(sown) // 2
|
| 106 |
+
for u in sown[:half] or sown[:1]:
|
| 107 |
+
cmds.append(C.attack_move([str(u["id"])], target_x=55, target_y=12))
|
| 108 |
+
for u in sown[half:]:
|
| 109 |
+
cmds.append(C.attack_move([str(u["id"])], target_x=55, target_y=28))
|
| 110 |
+
return cmds
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
# ── structural tests ────────────────────────────────────────────────
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def test_pack_loads_and_meta_reasoning():
|
| 117 |
+
pack = load_pack(PACK)
|
| 118 |
+
assert pack.meta.id == "scout-deny-enemy-vision"
|
| 119 |
+
assert pack.meta.capability == "reasoning"
|
| 120 |
+
assert pack.meta.real_world_meaning
|
| 121 |
+
assert pack.meta.robotics_analogue
|
| 122 |
+
anchors = " ".join(pack.meta.benchmark_anchor).lower()
|
| 123 |
+
assert "counter-reconnaissance" in anchors, anchors
|
| 124 |
+
assert "scout denial" in anchors, anchors
|
| 125 |
+
assert "intrusion prevention" in anchors, anchors
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def test_tools_are_combat_only():
|
| 129 |
+
pack = load_pack(PACK)
|
| 130 |
+
tools = set(pack.base.get("tools", []) if isinstance(pack.base, dict) else [])
|
| 131 |
+
for required in ("move_units", "attack_unit", "attack_move", "stop"):
|
| 132 |
+
assert required in tools, f"missing tool: {required!r}"
|
| 133 |
+
assert "build" not in tools, "this is a counter-recon combat pack — no build tool"
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def test_scouts_are_holdfire_e3():
|
| 137 |
+
"""Every level's enemy scouts are e3 at stance:0 (HoldFire) — they
|
| 138 |
+
hold an observation post and never advance or fight."""
|
| 139 |
+
pack = load_pack(PACK)
|
| 140 |
+
counts = {"easy": 1, "medium": 2, "hard": 2}
|
| 141 |
+
for lvl in LEVELS:
|
| 142 |
+
c = compile_level(pack, lvl)
|
| 143 |
+
scouts = [
|
| 144 |
+
a for a in c.scenario.actors if a.owner == "enemy" and a.type == "e3"
|
| 145 |
+
]
|
| 146 |
+
assert len(scouts) == counts[lvl], (
|
| 147 |
+
f"{lvl}: expected {counts[lvl]} e3 scouts; got {len(scouts)}"
|
| 148 |
+
)
|
| 149 |
+
for s in scouts:
|
| 150 |
+
assert s.stance == 0, f"{lvl}: scout must be stance:0 HoldFire"
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def test_agent_force_is_holdfire():
|
| 154 |
+
"""The agent strike force is stance:0 so it never auto-hunts —
|
| 155 |
+
a stall cannot clear the scouts for free."""
|
| 156 |
+
pack = load_pack(PACK)
|
| 157 |
+
for lvl in LEVELS:
|
| 158 |
+
c = compile_level(pack, lvl)
|
| 159 |
+
tanks = [
|
| 160 |
+
a for a in c.scenario.actors if a.owner == "agent" and a.type == "2tnk"
|
| 161 |
+
]
|
| 162 |
+
assert tanks, f"{lvl}: agent must have a 2tnk strike force"
|
| 163 |
+
for t in tanks:
|
| 164 |
+
assert t.stance == 0, f"{lvl}: agent tank must be stance:0 HoldFire"
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def test_kill_bar_matches_scout_count():
|
| 168 |
+
"""The win kill bar must equal the number of scouts — every
|
| 169 |
+
recon vector must be denied."""
|
| 170 |
+
pack = load_pack(PACK)
|
| 171 |
+
counts = {"easy": 1, "medium": 2, "hard": 2}
|
| 172 |
+
for lvl in LEVELS:
|
| 173 |
+
L = pack.levels[lvl]
|
| 174 |
+
bar = next(
|
| 175 |
+
int(c["units_killed_gte"])
|
| 176 |
+
for c in L.win_condition.model_dump()["all_of"]
|
| 177 |
+
if "units_killed_gte" in c
|
| 178 |
+
)
|
| 179 |
+
assert bar == counts[lvl], f"{lvl}: kill bar {bar} != scout count {counts[lvl]}"
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
def test_every_level_has_reachable_timeout_fail():
|
| 183 |
+
pack = load_pack(PACK)
|
| 184 |
+
for lvl in LEVELS:
|
| 185 |
+
L = pack.levels[lvl]
|
| 186 |
+
ceiling = 93 + 90 * (L.max_turns - 1)
|
| 187 |
+
wt = next(
|
| 188 |
+
int(c["within_ticks"])
|
| 189 |
+
for c in L.win_condition.model_dump()["all_of"]
|
| 190 |
+
if "within_ticks" in c
|
| 191 |
+
)
|
| 192 |
+
ft = next(
|
| 193 |
+
int(c["after_ticks"])
|
| 194 |
+
for c in L.fail_condition.model_dump()["any_of"]
|
| 195 |
+
if "after_ticks" in c
|
| 196 |
+
)
|
| 197 |
+
assert wt < ceiling, f"{lvl}: within_ticks {wt} >= ceiling {ceiling}"
|
| 198 |
+
assert ft <= ceiling, f"{lvl}: after_ticks {ft} > ceiling {ceiling}"
|
| 199 |
+
assert wt + 1 == ft, f"{lvl}: within/after mismatch {wt}/{ft}"
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def test_every_level_has_a_fail_condition():
|
| 203 |
+
pack = load_pack(PACK)
|
| 204 |
+
for lvl in LEVELS:
|
| 205 |
+
c = compile_level(pack, lvl)
|
| 206 |
+
assert c.fail_condition is not None, f"{lvl} needs a fail_condition"
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
def test_hard_has_two_seed_driven_spawn_groups():
|
| 210 |
+
c = compile_level(load_pack(PACK), "hard")
|
| 211 |
+
sp = {
|
| 212 |
+
(a.spawn_point if a.spawn_point is not None else 0)
|
| 213 |
+
for a in c.scenario.actors
|
| 214 |
+
if a.owner == "agent"
|
| 215 |
+
}
|
| 216 |
+
assert sp == {0, 1}, f"hard must define spawn_point groups {{0,1}}; got {sorted(sp)}"
|
| 217 |
+
|
| 218 |
+
|
| 219 |
+
def test_in_bounds_actors_on_every_level():
|
| 220 |
+
pack = load_pack(PACK)
|
| 221 |
+
for lvl in LEVELS:
|
| 222 |
+
c = compile_level(pack, lvl)
|
| 223 |
+
for a in c.scenario.actors:
|
| 224 |
+
x, y = a.position
|
| 225 |
+
assert 2 <= x <= 126 and 2 <= y <= 38, (
|
| 226 |
+
f"{lvl}: actor {a.type} at ({x},{y}) out of bounds"
|
| 227 |
+
)
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
# ── predicate-level (no engine) ─────────────────────────────────────
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def _ctx(*, tick=0, killed=0):
|
| 234 |
+
import types
|
| 235 |
+
|
| 236 |
+
sig = types.SimpleNamespace(
|
| 237 |
+
game_tick=tick,
|
| 238 |
+
units_killed=killed,
|
| 239 |
+
units_lost=0,
|
| 240 |
+
own_buildings=[],
|
| 241 |
+
own_building_types=set(),
|
| 242 |
+
enemies_seen_ids=set(),
|
| 243 |
+
enemy_buildings_seen_ids=set(),
|
| 244 |
+
)
|
| 245 |
+
return WinContext(signals=sig, render_state={"units_summary": []})
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
def test_predicates_enforce_kill_before_window():
|
| 249 |
+
pe = compile_level(load_pack(PACK), "easy")
|
| 250 |
+
# kill 1 scout in time → WIN
|
| 251 |
+
assert evaluate(pe.win_condition, _ctx(tick=600, killed=1))
|
| 252 |
+
# 0 kills → not win
|
| 253 |
+
assert not evaluate(pe.win_condition, _ctx(tick=600, killed=0))
|
| 254 |
+
# kill but past the window → not win
|
| 255 |
+
assert not evaluate(pe.win_condition, _ctx(tick=1101, killed=1))
|
| 256 |
+
# past after_ticks → fail
|
| 257 |
+
assert evaluate(pe.fail_condition, _ctx(tick=1101, killed=0))
|
| 258 |
+
# in-window → not fail
|
| 259 |
+
assert not evaluate(pe.fail_condition, _ctx(tick=600, killed=0))
|
| 260 |
+
|
| 261 |
+
pm = compile_level(load_pack(PACK), "medium")
|
| 262 |
+
# only 1 of 2 scouts killed → not win, and the deadline still fails
|
| 263 |
+
assert not evaluate(pm.win_condition, _ctx(tick=600, killed=1))
|
| 264 |
+
assert evaluate(pm.win_condition, _ctx(tick=600, killed=2))
|
| 265 |
+
assert evaluate(pm.fail_condition, _ctx(tick=1302, killed=1))
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
# ── engine-driven: every lazy/wrong policy LOSES, intended WINS ──────
|
| 269 |
+
|
| 270 |
+
|
| 271 |
+
@pytest.mark.parametrize("level", LEVELS)
|
| 272 |
+
@pytest.mark.parametrize("seed", SEEDS)
|
| 273 |
+
def test_stall_loses_every_tier_and_seed(level, seed):
|
| 274 |
+
c = compile_level(load_pack(PACK), level)
|
| 275 |
+
r = run_level(c, _stall, seed=seed)
|
| 276 |
+
assert r.outcome == "loss", (
|
| 277 |
+
f"{level}/seed{seed}: stall must LOSE; got {r.outcome} "
|
| 278 |
+
f"killed={r.signals.units_killed} tick={r.signals.game_tick}"
|
| 279 |
+
)
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
@pytest.mark.parametrize("level", LEVELS)
|
| 283 |
+
@pytest.mark.parametrize("seed", SEEDS)
|
| 284 |
+
def test_wrong_path_loses_every_tier_and_seed(level, seed):
|
| 285 |
+
c = compile_level(load_pack(PACK), level)
|
| 286 |
+
r = run_level(c, _wrong_path, seed=seed)
|
| 287 |
+
assert r.outcome == "loss", (
|
| 288 |
+
f"{level}/seed{seed}: wrong-path must LOSE; got {r.outcome} "
|
| 289 |
+
f"killed={r.signals.units_killed} tick={r.signals.game_tick}"
|
| 290 |
+
)
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
@pytest.mark.parametrize("level", ("medium", "hard"))
|
| 294 |
+
@pytest.mark.parametrize("seed", SEEDS)
|
| 295 |
+
def test_one_lane_loses_on_multi_vector_tiers(level, seed):
|
| 296 |
+
"""Committing the whole force to one recon vector kills only one
|
| 297 |
+
scout; the other completes its report → LOSS on medium/hard."""
|
| 298 |
+
c = compile_level(load_pack(PACK), level)
|
| 299 |
+
r = run_level(c, _one_lane, seed=seed)
|
| 300 |
+
assert r.outcome == "loss", (
|
| 301 |
+
f"{level}/seed{seed}: one-lane must LOSE; got {r.outcome} "
|
| 302 |
+
f"killed={r.signals.units_killed} tick={r.signals.game_tick}"
|
| 303 |
+
)
|
| 304 |
+
|
| 305 |
+
|
| 306 |
+
@pytest.mark.parametrize("level", LEVELS)
|
| 307 |
+
@pytest.mark.parametrize("seed", SEEDS)
|
| 308 |
+
def test_counter_recon_wins_every_tier_and_seed(level, seed):
|
| 309 |
+
"""The intended capability — cover EVERY recon vector — WINS
|
| 310 |
+
every tier and every hard seed."""
|
| 311 |
+
c = compile_level(load_pack(PACK), level)
|
| 312 |
+
r = run_level(c, _counter_recon, seed=seed)
|
| 313 |
+
assert r.outcome == "win", (
|
| 314 |
+
f"{level}/seed{seed}: counter-recon must WIN; got {r.outcome} "
|
| 315 |
+
f"killed={r.signals.units_killed} tick={r.signals.game_tick}"
|
| 316 |
+
)
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
def test_counter_recon_run_is_deterministic_per_seed():
|
| 320 |
+
c = compile_level(load_pack(PACK), "medium")
|
| 321 |
+
a = run_level(c, _counter_recon, seed=2)
|
| 322 |
+
b = run_level(c, _counter_recon, seed=2)
|
| 323 |
+
assert (a.outcome, a.turns, a.signals.units_killed) == (
|
| 324 |
+
b.outcome, b.turns, b.signals.units_killed
|
| 325 |
+
)
|