Yiyu Tian commited on
Commit
42e070b
·
1 Parent(s): d3514c7

P1: fail_conditions + new longhorizon scenario

Browse files

- rush-hour easy/medium: add units_lost_lte caps and timeout/force-
destruction fail_conditions (was: kill-3-in-16k-ticks auto-win with
~32 stance-3 friendly units; now caps lost at 6/4 and loses on
timeout). Hard gains a timeout-fail alongside its force-destruction
fail.
- economy-investment: add fail_condition to all 3 levels (each loses
on timeout; hard additionally loses on exceeding the units_lost_lte:0
cap). Restores 3-way outcomes; run_level otherwise collapses
unmet-win into draw.
- new pack longhorizon-opening-to-assault: scout -> tech -> army ->
strike chained in a single 24k-40k-tick episode with terminal-only
all_of credit and timeout-fail. Fills the long-horizon
credit-assignment gap (Terminal-Bench / OSWorld 50-step analogue)
using only existing predicates.

All edits validate against the win-condition pydantic schema.

openra_bench/scenarios/packs/economy-investment.yaml CHANGED
@@ -74,6 +74,8 @@ levels:
74
  - building_count_gte: {type: proc, n: 2}
75
  - building_count_gte: {type: powr, n: 2}
76
  - within_ticks: 22000
 
 
77
  max_turns: 70
78
  medium:
79
  description: >
@@ -88,6 +90,8 @@ levels:
88
  - building_count_gte: {type: powr, n: 2}
89
  - own_units_gte: 3
90
  - within_ticks: 16000
 
 
91
  max_turns: 80
92
  hard:
93
  description: >
@@ -102,4 +106,8 @@ levels:
102
  - building_count_gte: {type: proc, n: 1}
103
  - units_lost_lte: 0
104
  - within_ticks: 14000
 
 
 
 
105
  max_turns: 90
 
74
  - building_count_gte: {type: proc, n: 2}
75
  - building_count_gte: {type: powr, n: 2}
76
  - within_ticks: 22000
77
+ fail_condition:
78
+ after_ticks: 22001
79
  max_turns: 70
80
  medium:
81
  description: >
 
90
  - building_count_gte: {type: powr, n: 2}
91
  - own_units_gte: 3
92
  - within_ticks: 16000
93
+ fail_condition:
94
+ after_ticks: 16001
95
  max_turns: 80
96
  hard:
97
  description: >
 
106
  - building_count_gte: {type: proc, n: 1}
107
  - units_lost_lte: 0
108
  - within_ticks: 14000
109
+ fail_condition:
110
+ any_of:
111
+ - after_ticks: 14001
112
+ - not: {units_lost_lte: 0}
113
  max_turns: 90
openra_bench/scenarios/packs/longhorizon-opening-to-assault.yaml ADDED
@@ -0,0 +1,149 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Validate: python -m openra_bench.scenarios.validate \
2
+ # openra_bench/scenarios/packs/longhorizon-opening-to-assault.yaml
3
+ # See ../CONTRIBUTING.md for the full win-condition grammar.
4
+ #
5
+ # LONG-HORIZON CREDIT ASSIGNMENT focus. Every other pack is a short
6
+ # single-phase test (one decision, one clock). This one chains four
7
+ # phases in a single episode with a TERMINAL-ONLY win predicate so the
8
+ # score depends on every earlier choice still being correct at the end:
9
+ # scout -> the map is foggy and the agent must establish coverage
10
+ # tech -> a refinery must exist (the tech-graph waypoint)
11
+ # army -> a real combat force must be standing
12
+ # strike -> a contingent of that force must reach the enemy base
13
+ # The win is `all_of` of these four — partial completion = loss on
14
+ # timeout, draws are impossible. Skipping scouting still passes only if
15
+ # the model bulldozed through fog (rare); skipping tech is hard-failing.
16
+ # This is the OpenRA analogue of long-horizon agentic benchmarks
17
+ # (Terminal-Bench 2.0, OSWorld 50-step, GAIA multi-hop): credit must
18
+ # survive the full opening -> tech -> army -> assault chain.
19
+
20
+ meta:
21
+ id: longhorizon-opening-to-assault
22
+ title: 'Opening-to-Assault — Chain Scout, Tech, Army, and Strike in One Episode'
23
+ capability: reasoning
24
+ real_world_meaning: >
25
+ A long single-episode plan whose terminal score depends on every
26
+ intermediate phase being completed in order: scout the map, stand
27
+ up the supporting tech, build a real fighting force, and commit it
28
+ to a positional objective on the enemy base. Partial progress is
29
+ not a partial score; skipping or delaying any phase fails the win.
30
+ robotics_analogue: >
31
+ Long-horizon autonomous operation: a mission that must execute
32
+ distinct phases (sensor coverage, infrastructure, fleet
33
+ provisioning, terminal action) within one budget window, where
34
+ credit is only assigned at the end of the episode.
35
+ author: openra-bench
36
+
37
+ base_map: rush-hour-arena
38
+
39
+ base:
40
+ agent:
41
+ faction: allies
42
+ cash: 0
43
+ enemy:
44
+ faction: soviet
45
+ cash: 0
46
+ bot_type: ''
47
+ tools:
48
+ - build
49
+ - place_building
50
+ - move_units
51
+ - attack_unit
52
+ - deploy
53
+ - stop_units
54
+ planning: true
55
+ termination:
56
+ max_ticks: 60000
57
+ actors:
58
+ # Pre-seeded agent base in the western corner: a construction yard,
59
+ # one power plant, and a barracks (tech prerequisite for infantry).
60
+ # The agent must still build proc (refinery) + recruit a force +
61
+ # cross the map.
62
+ - type: fact
63
+ owner: agent
64
+ position: [10, 18]
65
+ - type: powr
66
+ owner: agent
67
+ position: [14, 18]
68
+ - type: tent
69
+ owner: agent
70
+ position: [10, 22]
71
+ # Static enemy buildings on the far east mark the assault target.
72
+ - type: fact
73
+ owner: enemy
74
+ position: [115, 30]
75
+ - type: proc
76
+ owner: enemy
77
+ position: [115, 34]
78
+ # A few defenders so reach_region is not free.
79
+ - type: e1
80
+ owner: enemy
81
+ position: [108, 30]
82
+ stance: 2
83
+ count: 2
84
+ - type: e3
85
+ owner: enemy
86
+ position: [108, 34]
87
+ stance: 2
88
+
89
+ starting_cash: 5000
90
+
91
+ levels:
92
+ easy:
93
+ description: >
94
+ Generous clock (40k ticks), generous coverage bar. Chain the four
95
+ phases in order: explore >=35% of the map, build a refinery
96
+ (proc), stand up >=4 combat units, and put at least one unit on
97
+ the enemy base at (115, 32). Missing any of the four at episode
98
+ end is a loss, not a draw.
99
+ starting_cash: 5000
100
+ win_condition:
101
+ all_of:
102
+ - explored_pct_gte: 35
103
+ - has_building: proc
104
+ - own_units_gte: 4
105
+ - reach_region: {x: 115, y: 32, radius: 8}
106
+ - within_ticks: 40000
107
+ fail_condition:
108
+ after_ticks: 40001
109
+ max_turns: 120
110
+
111
+ medium:
112
+ description: >
113
+ Tighter clock (32k ticks), coverage 45%, force >=5, refinery
114
+ required, contact the enemy base. Same chain; less slack.
115
+ starting_cash: 5000
116
+ win_condition:
117
+ all_of:
118
+ - explored_pct_gte: 45
119
+ - has_building: proc
120
+ - own_units_gte: 5
121
+ - reach_region: {x: 115, y: 32, radius: 7}
122
+ - within_ticks: 32000
123
+ fail_condition:
124
+ after_ticks: 32001
125
+ max_turns: 140
126
+
127
+ hard:
128
+ description: >
129
+ Short clock (24k ticks), coverage 55%, force >=6, refinery
130
+ required, both an enemy building discovered AND a unit on the
131
+ enemy base. An attrition cap prevents brute-force loss-then-win.
132
+ Skipping the scouting phase to rush tech-then-strike fails the
133
+ coverage clause; skipping tech to rush units fails the
134
+ has_building clause.
135
+ starting_cash: 4500
136
+ win_condition:
137
+ all_of:
138
+ - explored_pct_gte: 55
139
+ - buildings_discovered_gte: 1
140
+ - has_building: proc
141
+ - own_units_gte: 6
142
+ - reach_region: {x: 115, y: 32, radius: 6}
143
+ - within_ticks: 24000
144
+ - units_lost_lte: 4
145
+ fail_condition:
146
+ any_of:
147
+ - after_ticks: 24001
148
+ - not: {own_units_gte: 1}
149
+ max_turns: 160
openra_bench/scenarios/packs/rush-hour.yaml CHANGED
@@ -516,27 +516,45 @@ base:
516
  enemy_units_killed: true
517
  levels:
518
  easy:
519
- description: "Rush Hour Sweep \u2014 easy difficulty (ported)."
 
 
 
520
  win_condition:
521
  all_of:
522
- - units_killed_gte: 3
523
  - within_ticks: 16000
 
 
 
 
 
524
  medium:
525
- description: "Rush Hour Sweep \u2014 medium difficulty (ported)."
 
 
 
526
  win_condition:
527
  all_of:
528
- - units_killed_gte: 6
529
  - within_ticks: 13000
 
 
 
 
 
530
  hard:
531
  description: >
532
  Rush Hour Sweep \u2014 hard. The agent force already round-robins
533
  between two base spawn_point groups by seed (start corner varies,
534
  anti-memorisation); hard adds a tight clock, an attrition cap,
535
- and a real loss on force destruction.
536
  win_condition:
537
  all_of:
538
  - units_killed_gte: 9
539
  - within_ticks: 10000
540
  - units_lost_lte: 3
541
  fail_condition:
542
- not: {own_units_gte: 1}
 
 
 
516
  enemy_units_killed: true
517
  levels:
518
  easy:
519
+ description: >
520
+ Rush Hour Sweep \u2014 easy. Find and destroy >=5 scattered enemy
521
+ units within the time budget, losing <=6 of your own. Timeout
522
+ or total force destruction is a loss, not a draw.
523
  win_condition:
524
  all_of:
525
+ - units_killed_gte: 5
526
  - within_ticks: 16000
527
+ - units_lost_lte: 6
528
+ fail_condition:
529
+ any_of:
530
+ - not: {own_units_gte: 1}
531
+ - after_ticks: 16001
532
  medium:
533
+ description: >
534
+ Rush Hour Sweep \u2014 medium. Tighter clock and attrition cap:
535
+ destroy >=7 enemy units in 13000 ticks, losing <=4. Timeout is
536
+ a loss.
537
  win_condition:
538
  all_of:
539
+ - units_killed_gte: 7
540
  - within_ticks: 13000
541
+ - units_lost_lte: 4
542
+ fail_condition:
543
+ any_of:
544
+ - not: {own_units_gte: 1}
545
+ - after_ticks: 13001
546
  hard:
547
  description: >
548
  Rush Hour Sweep \u2014 hard. The agent force already round-robins
549
  between two base spawn_point groups by seed (start corner varies,
550
  anti-memorisation); hard adds a tight clock, an attrition cap,
551
+ and a real loss on force destruction or timeout.
552
  win_condition:
553
  all_of:
554
  - units_killed_gte: 9
555
  - within_ticks: 10000
556
  - units_lost_lte: 3
557
  fail_condition:
558
+ any_of:
559
+ - not: {own_units_gte: 1}
560
+ - after_ticks: 10001