yxc20098 commited on
Commit
1400f05
·
1 Parent(s): ca00016

docs: CLAUDE.md + AGENTS.md — pointer for friend's Claude Code / Codex agents to use the no-cheat scenario methodology

Browse files

Claude Code auto-loads CLAUDE.md from repo root; Codex/general-purpose
agents follow AGENTS.md by convention. Both point at the authoritative
SCENARIO_REVIEW_CHECKLIST.md (the closer-look A/B/C methodology) and
SCENARIO_QUALITY.md (whole-suite no-cheat pass + engine footguns), plus
distil the bar ('no defect, no cheat'), the predicate-/phrase-pairing
rule, the engine facts agents must internalise (90 ticks/turn, tick
formula, engine auto-done on enemy elimination → fact marker), the
known engine footguns (power_surplus_gte inert; deploy unimplemented),
how to validate deterministically via scripted policies through
openra_bench.eval_core.run_level (no model / no network), and the
commit-on-main authorisation policy plus the no-Claude-co-author rule.

Files changed (2) hide show
  1. AGENTS.md +8 -0
  2. CLAUDE.md +127 -0
AGENTS.md ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ # AGENTS.md — OpenRA-Bench
2
+
3
+ Codex and other coding agents: the project's agent guide is **`CLAUDE.md`**
4
+ in this same directory. Read it first — it is identical content,
5
+ maintained as the single source of truth for any AI assistant working
6
+ in this repo. The full methodology lives in
7
+ `SCENARIO_REVIEW_CHECKLIST.md` and the worked examples are the
8
+ no-cheat-redesign commits on `main`.
CLAUDE.md ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Working in OpenRA-Bench (Claude Code / Codex / any coding-agent guide)
2
+
3
+ OpenRA-Bench is a rigorous LLM-agent RTS benchmark on a Rust engine. If
4
+ you are an AI coding agent creating, validating, fixing, or extending a
5
+ scenario pack, read this once and treat the linked docs as binding.
6
+
7
+ ## The bar (apply to every scenario you touch)
8
+
9
+ > **No defect. No cheat.**
10
+ > Every lazy / brute / stall / blind / shortest-path / wrong-route /
11
+ > spam policy must **LOSE** on every level and every hard seed (1–4).
12
+ > The intended capability policy must **WIN**. Non-win must be a real
13
+ > reachable timeout **LOSS** — never a draw.
14
+
15
+ A scenario is defective if any of the following hold:
16
+
17
+ 1. The win predicate is satisfiable by a play that ignores the
18
+ advertised capability (the "laziest play wins" inversion).
19
+ 2. `within_ticks` or `after_ticks` is set above the tick reachable
20
+ within `max_turns` (engine advances ~90 ticks per decision turn ⇒
21
+ `tick ≤ 93 + 90·(max_turns − 1)`); the deadline never bites ⇒ the
22
+ episode times out as a **DRAW**, not a LOSS.
23
+ 3. There is no `fail_condition`, or it only triggers on full
24
+ force-wipe; a stall / preserve / partial outcome silently draws.
25
+ 4. The intended capability is not solvable inside the declared budget
26
+ (a scenario nobody can win is also defective).
27
+ 5. The engine auto-terminates on enemy-elimination before the win/fail
28
+ is evaluated (mitigation: place an unarmed high-HP enemy `fact`
29
+ marker at the objective).
30
+ 6. Actors are placed outside the map's playable bounds (engine
31
+ panics).
32
+ 7. The pack is `UPGRADED` in `tests/test_hard_tier.py` but its hard
33
+ tier does not produce ≥2 distinct seed-driven spawns (or there is
34
+ no documented `NOT_APPLICABLE` reason).
35
+
36
+ ## Authoritative docs (read these, in order)
37
+
38
+ - **`SCENARIO_REVIEW_CHECKLIST.md`** — the closer-look methodology
39
+ (A solvency / B stability / C capability) you follow step by step
40
+ to create or validate a pack.
41
+ - **`SCENARIO_QUALITY.md`** — the whole-suite no-cheat pass summary,
42
+ the recurring defect classes the pass eliminated, and the
43
+ predicate-idiom recipe (which predicate makes which capability
44
+ load-bearing), plus engine footguns to avoid.
45
+ - **`openra_bench/scenarios/win_conditions.py`** — the predicate
46
+ grammar. If you add a new predicate, you **must** also add a
47
+ `_PHRASES` / `_REGION_PHRASES` translation in
48
+ `openra_bench/game_knowledge.py` (the suite test
49
+ `test_all_predicate_keys_have_a_translation` enforces this).
50
+ - **`openra_bench/botgen.py`** + `openra-sim/src/scripted_bot.rs` —
51
+ the scripted opponents (`hunt | rusher | patrol | turtle | guard`)
52
+ declared per-pack as `enemy: {bot_type: <name>}`. `guard` is the
53
+ leashed defender used by the bait/decoy/lure idioms.
54
+ - **The 21 no-cheat-redesign commits on `main`** are worked examples
55
+ of every capability/predicate/bot combination. Browse with
56
+ `git log --oneline --grep "no-cheat redesign"` and read the bodies.
57
+
58
+ ## Engine facts you must internalise
59
+
60
+ - **Ticks/turn:** ~90. Max tick at `max_turns` ≈ `93 + 90·(max_turns
61
+ − 1)`. Any `within_ticks` / `after_ticks` above this is **inert**
62
+ (won't bite) ⇒ draw degeneracy.
63
+ - **Engine auto-done:** the engine sets `done=True` when all enemy
64
+ actors are eliminated, or sometimes when an agent unit reaches an
65
+ enemy-key location. Without a persistent enemy actor a win-by-reach
66
+ scenario can end as DRAW. Put an unarmed high-HP enemy `fact`
67
+ marker at the objective.
68
+ - **Own-unit `actor_type`** surfaces in `units_summary`
69
+ (`unit_type_count_eq / _gte` work). Predicates relying on it are
70
+ valid.
71
+ - **`power_surplus_gte` is currently inert** (obs reports
72
+ `power_provided / power_drained = 0`). Do **not** rely on it as a
73
+ sole discriminator.
74
+ - **`deploy` is unimplemented** in the installed wheel (MCV never
75
+ becomes a Construction Yard). Build-radius creep is the supported
76
+ alternative.
77
+ - **Scripted bot `guard`:** holds its post (`spawn_cell`), auto-fires
78
+ in range, lunges at the nearest foe within `GUARD_AGGRO ≈ 16`,
79
+ snaps back past `GUARD_LEASH ≈ 18` — the bait-able-defender idiom
80
+ proven in #4 / #6 / #7 / #15 / #18.
81
+
82
+ ## How to validate (deterministic, no model / no network)
83
+
84
+ For each level + each hard seed (1–4) run scripted policies via
85
+ `openra_bench.eval_core.run_level`:
86
+
87
+ ```python
88
+ from openra_bench.scenarios import load_pack
89
+ from openra_bench.scenarios.loader import PACKS_DIR, compile_level
90
+ from openra_bench.eval_core import run_level
91
+
92
+ c = compile_level(load_pack(PACKS_DIR / "<pack>.yaml"), "easy")
93
+ res = run_level(c, my_policy_fn, seed=1)
94
+ print(res.outcome, res.turns, res.signals.units_lost)
95
+ ```
96
+
97
+ Cover at minimum: `stall` (only `Command.observe()`), a `brute`
98
+ beeline to the objective, a `greedy` / `wrong-path` if applicable,
99
+ and the **intended** capability policy. The bar (above) must hold.
100
+ **No model / OpenRouter / network runs are needed for validation** —
101
+ scripted policies are sufficient and free.
102
+
103
+ ## Working on `main` (PRs vs direct push)
104
+
105
+ - The default branch is `main`.
106
+ - Direct pushes to `main` are reserved for the user's batch
107
+ parallel-agent workflow. If you are a one-off agent invoked
108
+ outside that flow, **branch first** and open a PR; do not push to
109
+ `main` without explicit user authorization.
110
+ - Commits must NOT include a Claude / AI co-author line.
111
+ - The shared engine wheel is rebuilt via
112
+ `cd OpenRA-Rust && PATH=$HOME/.cargo/bin:/opt/anaconda3/bin:$PATH
113
+ maturin develop --release` — **verify the `Installed openra_train`
114
+ line actually printed** (maturin can exit 0 while cargo failed).
115
+
116
+ ## Don'ts (lessons from the cadence)
117
+
118
+ - Don't add a predicate to `win_conditions.py` without a
119
+ `_PHRASES` translation in `game_knowledge.py`.
120
+ - Don't `git add -A` / `git commit -a` — concurrent agents may have
121
+ uncommitted edits to shared files; stage only your own files.
122
+ - Don't compensate for model weakness or over-engineer; only fix
123
+ real scenario defects, and keep the established idioms.
124
+ - Don't edit `SCENARIO_QUALITY.md` / `docs/scenarios.html` in a
125
+ per-scenario commit — the main session regenerates them at the end.
126
+ - Don't edit `OpenRA-Rust/` (the engine) inside a scenario task —
127
+ flag engine needs in your report instead.