Spaces:
Running
Running
docs: CLAUDE.md + AGENTS.md — pointer for friend's Claude Code / Codex agents to use the no-cheat scenario methodology
Browse filesClaude Code auto-loads CLAUDE.md from repo root; Codex/general-purpose
agents follow AGENTS.md by convention. Both point at the authoritative
SCENARIO_REVIEW_CHECKLIST.md (the closer-look A/B/C methodology) and
SCENARIO_QUALITY.md (whole-suite no-cheat pass + engine footguns), plus
distil the bar ('no defect, no cheat'), the predicate-/phrase-pairing
rule, the engine facts agents must internalise (90 ticks/turn, tick
formula, engine auto-done on enemy elimination → fact marker), the
known engine footguns (power_surplus_gte inert; deploy unimplemented),
how to validate deterministically via scripted policies through
openra_bench.eval_core.run_level (no model / no network), and the
commit-on-main authorisation policy plus the no-Claude-co-author rule.
AGENTS.md
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# AGENTS.md — OpenRA-Bench
|
| 2 |
+
|
| 3 |
+
Codex and other coding agents: the project's agent guide is **`CLAUDE.md`**
|
| 4 |
+
in this same directory. Read it first — it is identical content,
|
| 5 |
+
maintained as the single source of truth for any AI assistant working
|
| 6 |
+
in this repo. The full methodology lives in
|
| 7 |
+
`SCENARIO_REVIEW_CHECKLIST.md` and the worked examples are the
|
| 8 |
+
no-cheat-redesign commits on `main`.
|
CLAUDE.md
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Working in OpenRA-Bench (Claude Code / Codex / any coding-agent guide)
|
| 2 |
+
|
| 3 |
+
OpenRA-Bench is a rigorous LLM-agent RTS benchmark on a Rust engine. If
|
| 4 |
+
you are an AI coding agent creating, validating, fixing, or extending a
|
| 5 |
+
scenario pack, read this once and treat the linked docs as binding.
|
| 6 |
+
|
| 7 |
+
## The bar (apply to every scenario you touch)
|
| 8 |
+
|
| 9 |
+
> **No defect. No cheat.**
|
| 10 |
+
> Every lazy / brute / stall / blind / shortest-path / wrong-route /
|
| 11 |
+
> spam policy must **LOSE** on every level and every hard seed (1–4).
|
| 12 |
+
> The intended capability policy must **WIN**. Non-win must be a real
|
| 13 |
+
> reachable timeout **LOSS** — never a draw.
|
| 14 |
+
|
| 15 |
+
A scenario is defective if any of the following hold:
|
| 16 |
+
|
| 17 |
+
1. The win predicate is satisfiable by a play that ignores the
|
| 18 |
+
advertised capability (the "laziest play wins" inversion).
|
| 19 |
+
2. `within_ticks` or `after_ticks` is set above the tick reachable
|
| 20 |
+
within `max_turns` (engine advances ~90 ticks per decision turn ⇒
|
| 21 |
+
`tick ≤ 93 + 90·(max_turns − 1)`); the deadline never bites ⇒ the
|
| 22 |
+
episode times out as a **DRAW**, not a LOSS.
|
| 23 |
+
3. There is no `fail_condition`, or it only triggers on full
|
| 24 |
+
force-wipe; a stall / preserve / partial outcome silently draws.
|
| 25 |
+
4. The intended capability is not solvable inside the declared budget
|
| 26 |
+
(a scenario nobody can win is also defective).
|
| 27 |
+
5. The engine auto-terminates on enemy-elimination before the win/fail
|
| 28 |
+
is evaluated (mitigation: place an unarmed high-HP enemy `fact`
|
| 29 |
+
marker at the objective).
|
| 30 |
+
6. Actors are placed outside the map's playable bounds (engine
|
| 31 |
+
panics).
|
| 32 |
+
7. The pack is `UPGRADED` in `tests/test_hard_tier.py` but its hard
|
| 33 |
+
tier does not produce ≥2 distinct seed-driven spawns (or there is
|
| 34 |
+
no documented `NOT_APPLICABLE` reason).
|
| 35 |
+
|
| 36 |
+
## Authoritative docs (read these, in order)
|
| 37 |
+
|
| 38 |
+
- **`SCENARIO_REVIEW_CHECKLIST.md`** — the closer-look methodology
|
| 39 |
+
(A solvency / B stability / C capability) you follow step by step
|
| 40 |
+
to create or validate a pack.
|
| 41 |
+
- **`SCENARIO_QUALITY.md`** — the whole-suite no-cheat pass summary,
|
| 42 |
+
the recurring defect classes the pass eliminated, and the
|
| 43 |
+
predicate-idiom recipe (which predicate makes which capability
|
| 44 |
+
load-bearing), plus engine footguns to avoid.
|
| 45 |
+
- **`openra_bench/scenarios/win_conditions.py`** — the predicate
|
| 46 |
+
grammar. If you add a new predicate, you **must** also add a
|
| 47 |
+
`_PHRASES` / `_REGION_PHRASES` translation in
|
| 48 |
+
`openra_bench/game_knowledge.py` (the suite test
|
| 49 |
+
`test_all_predicate_keys_have_a_translation` enforces this).
|
| 50 |
+
- **`openra_bench/botgen.py`** + `openra-sim/src/scripted_bot.rs` —
|
| 51 |
+
the scripted opponents (`hunt | rusher | patrol | turtle | guard`)
|
| 52 |
+
declared per-pack as `enemy: {bot_type: <name>}`. `guard` is the
|
| 53 |
+
leashed defender used by the bait/decoy/lure idioms.
|
| 54 |
+
- **The 21 no-cheat-redesign commits on `main`** are worked examples
|
| 55 |
+
of every capability/predicate/bot combination. Browse with
|
| 56 |
+
`git log --oneline --grep "no-cheat redesign"` and read the bodies.
|
| 57 |
+
|
| 58 |
+
## Engine facts you must internalise
|
| 59 |
+
|
| 60 |
+
- **Ticks/turn:** ~90. Max tick at `max_turns` ≈ `93 + 90·(max_turns
|
| 61 |
+
− 1)`. Any `within_ticks` / `after_ticks` above this is **inert**
|
| 62 |
+
(won't bite) ⇒ draw degeneracy.
|
| 63 |
+
- **Engine auto-done:** the engine sets `done=True` when all enemy
|
| 64 |
+
actors are eliminated, or sometimes when an agent unit reaches an
|
| 65 |
+
enemy-key location. Without a persistent enemy actor a win-by-reach
|
| 66 |
+
scenario can end as DRAW. Put an unarmed high-HP enemy `fact`
|
| 67 |
+
marker at the objective.
|
| 68 |
+
- **Own-unit `actor_type`** surfaces in `units_summary`
|
| 69 |
+
(`unit_type_count_eq / _gte` work). Predicates relying on it are
|
| 70 |
+
valid.
|
| 71 |
+
- **`power_surplus_gte` is currently inert** (obs reports
|
| 72 |
+
`power_provided / power_drained = 0`). Do **not** rely on it as a
|
| 73 |
+
sole discriminator.
|
| 74 |
+
- **`deploy` is unimplemented** in the installed wheel (MCV never
|
| 75 |
+
becomes a Construction Yard). Build-radius creep is the supported
|
| 76 |
+
alternative.
|
| 77 |
+
- **Scripted bot `guard`:** holds its post (`spawn_cell`), auto-fires
|
| 78 |
+
in range, lunges at the nearest foe within `GUARD_AGGRO ≈ 16`,
|
| 79 |
+
snaps back past `GUARD_LEASH ≈ 18` — the bait-able-defender idiom
|
| 80 |
+
proven in #4 / #6 / #7 / #15 / #18.
|
| 81 |
+
|
| 82 |
+
## How to validate (deterministic, no model / no network)
|
| 83 |
+
|
| 84 |
+
For each level + each hard seed (1–4) run scripted policies via
|
| 85 |
+
`openra_bench.eval_core.run_level`:
|
| 86 |
+
|
| 87 |
+
```python
|
| 88 |
+
from openra_bench.scenarios import load_pack
|
| 89 |
+
from openra_bench.scenarios.loader import PACKS_DIR, compile_level
|
| 90 |
+
from openra_bench.eval_core import run_level
|
| 91 |
+
|
| 92 |
+
c = compile_level(load_pack(PACKS_DIR / "<pack>.yaml"), "easy")
|
| 93 |
+
res = run_level(c, my_policy_fn, seed=1)
|
| 94 |
+
print(res.outcome, res.turns, res.signals.units_lost)
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
Cover at minimum: `stall` (only `Command.observe()`), a `brute`
|
| 98 |
+
beeline to the objective, a `greedy` / `wrong-path` if applicable,
|
| 99 |
+
and the **intended** capability policy. The bar (above) must hold.
|
| 100 |
+
**No model / OpenRouter / network runs are needed for validation** —
|
| 101 |
+
scripted policies are sufficient and free.
|
| 102 |
+
|
| 103 |
+
## Working on `main` (PRs vs direct push)
|
| 104 |
+
|
| 105 |
+
- The default branch is `main`.
|
| 106 |
+
- Direct pushes to `main` are reserved for the user's batch
|
| 107 |
+
parallel-agent workflow. If you are a one-off agent invoked
|
| 108 |
+
outside that flow, **branch first** and open a PR; do not push to
|
| 109 |
+
`main` without explicit user authorization.
|
| 110 |
+
- Commits must NOT include a Claude / AI co-author line.
|
| 111 |
+
- The shared engine wheel is rebuilt via
|
| 112 |
+
`cd OpenRA-Rust && PATH=$HOME/.cargo/bin:/opt/anaconda3/bin:$PATH
|
| 113 |
+
maturin develop --release` — **verify the `Installed openra_train`
|
| 114 |
+
line actually printed** (maturin can exit 0 while cargo failed).
|
| 115 |
+
|
| 116 |
+
## Don'ts (lessons from the cadence)
|
| 117 |
+
|
| 118 |
+
- Don't add a predicate to `win_conditions.py` without a
|
| 119 |
+
`_PHRASES` translation in `game_knowledge.py`.
|
| 120 |
+
- Don't `git add -A` / `git commit -a` — concurrent agents may have
|
| 121 |
+
uncommitted edits to shared files; stage only your own files.
|
| 122 |
+
- Don't compensate for model weakness or over-engineer; only fix
|
| 123 |
+
real scenario defects, and keep the established idioms.
|
| 124 |
+
- Don't edit `SCENARIO_QUALITY.md` / `docs/scenarios.html` in a
|
| 125 |
+
per-scenario commit — the main session regenerates them at the end.
|
| 126 |
+
- Don't edit `OpenRA-Rust/` (the engine) inside a scenario task —
|
| 127 |
+
flag engine needs in your report instead.
|