EVAL_STACK_PLAN.md

OpenRA-Bench → Unified Eval Stack Plan

Goal

Turn OpenRA-Bench into an efficient, customizable harness to evaluate model performance on spatial, multi-modal, and complex multi-target multi-step reasoning / planning, by reusing the mature eval stack from OpenRA-RL-Training on top of the Rust environment (OpenRA-Rust).

Two evaluation modes:

1. Fixed-scenario eval — N episodes of one model on a controlled scenario, scored with verifiable rubrics + composite score + 95% CIs.
2. Pairwise adversarial 1v1 — two models in one 1v1 map, win-rate / Elo.

Scenarios must be controlled and isolate specific capabilities (e.g. the observed failure: model cannot connect an unexplored region to a target area to explore — a vision/perception + spatial-planning deficiency).

Core framing: the Perception → Reasoning → Action chain

Every scenario evaluates one chain: read the (visual + symbolic) state → form a multi-step plan → emit valid commands that execute it. The eval must attribute failure to a specific link, not just report a score:

• Perception — did the model correctly read the minimap/state? (e.g. locate the unexplored region and the target). Probe via state-readback / forced-choice checks derived from ground-truth obs.
• Reasoning — given correct perception, did it form a valid plan? (e.g. connect unexplored→target, sequence multi-target order). Probe via plan quality vs optimal (path length, target ordering, sub-goal coverage).
• Action — did it emit syntactically/semantically valid commands that realize the plan? Probe via action-validity rate and plan↔execution drift.

Per-scenario rubrics carry one diagnostic per link so a low score points at the broken link. This is the primary product differentiator vs a raw win-rate bench.

Decisions (locked)

• Repo: refactor OpenRA-Bench in place; it imports Training's eval stack. OpenRA-RL-Training stays source of truth for the engine code.
• Backend: Rust only (openra_train PyO3). C# (openra-rl) is slow/fragile and is dropped from the eval path. Rust must be made faithful to the C# reference where scenarios require it.
• Multi-modal: reuse Training's minimap_renderer.render_minimap() PNG, injected as image_url in the agent prompt.

Source components reused from OpenRA-RL-Training

Component	Path	Role in Bench
Episode engine	openra_rl_training/training/agent_rollout.py (play_episodes_async)	Drives the real model loop (currently Bench's agent fn is a no-op)
Reward dims	training/reward_funcs.py	Per-scenario weighted scoring
Rust pool	training/rust_env_pool.py	The only backend
Minimap	training/minimap_renderer.py	Multi-modal observation
Scenarios/rubrics	scenarios/*.yaml, curricula/*.yaml, verifiable metrics	Controlled tasks + pass/fail
CI comparison	scripts/build_eval_comparison.py	Stat-sound model comparison

Rust faithfulness gap (drives sequencing)

• Commands: 3/22 (Move, Attack, Observe). Missing: Build/Train/Harvest/Deploy/ Sell/Repair/Stance/Transport/Power/RallyPoint/Guard/Patrol…
• Observations: ~30% of C# proto. Missing: economy, production, military stats, spatial tensor, kill_events, result/reward fields.
• Scenarios: 2 hand-built (rush-hour, scout-maginot). No generic .oramap load.
• Engine: movement (A*), combat, projectiles, fog, static defenses = done. Economy, production/tech, transport, multi-armament = not done.

→ Movement/combat/fog scenarios + combat-only 1v1 work today. Economy / production / tech scenarios require Rust engine work first.

CRITICAL FINDING (verified end-to-end, local)

play_episodes_async (agent_rollout.py:4815) is hardwired to the C# gRPC server via openra_env.mcp_ws_client.OpenRAMCPClient and is entangled with TRL (tokenizer, prompt_ids/completion_ids, worker pool, partial cache). It is not reusable as-is on Rust. rust_env_pool is used only by the lighter rollout.py path.

Also: minimap_renderer.render_minimap() expects state["minimap"] (ASCII), units_summary, enemy_summary — none of which the Rust env emits.

Verified live Rust obs schema (openra_train rush-hour, local wheel, Python 3.12, anaconda):

keys = enemy_buildings_summary, enemy_hp, enemy_positions, explored_cells,
       explored_percent, game_tick, unit_hp, unit_positions, units_killed
unit_positions = {actor_id: {cell_x, cell_y[, target, activity, ...]}}
step() -> (obs, reward=0.0 hardcoded, done:bool, info={game_tick, warnings})

No minimap ASCII, no economy/military/result/reward, no terrain.

Consequence: Phase 0 builds a Bench-side episode loop that reuses components (reward_funcs, minimap_renderer, scenario loader, action parser) behind a Rust→schema adapter (openra_bench/rust_adapter.py). The adapter is the crux and overlaps directly with the "make Rust faithful" workstream:

• unit_positions → units_summary (renderer/prompt schema)
• enemy_positions + enemy_buildings_summary → enemy_summary
• synthesize ASCII minimap from explored_cells + scenario map dims
• load terrain_png from the scenario's base .oramap (as Training does)
• derive scoring signals (kills, discovery, exploration, outcome) from obs deltas since Rust reward is hardcoded 0.0 — feeds reward_funcs + the P/R/A diagnostics directly.

Target Bench layout

OpenRA-Bench/
  openra_bench/
    eval_core.py      # thin wrapper over play_episodes_async, Rust backend forced
    agent.py          # REAL model agent (OpenAI-compatible), minimap multimodal
    scenarios/        # controlled eval scenarios (symlink/copy + Bench-authored)
    rubrics.py        # verifiable + composite scoring (reuse Training)
    pairwise.py       # 1v1 adversarial orchestration + Elo
  evaluate.py         # fixed-scenario CLI (rewritten to use eval_core)
  compare.py          # CI comparison front-end (wraps build_eval_comparison)
  app.py              # leaderboard (fed by both modes)

Phases

Phase 0 — Integration spine (no Rust changes)

• Bench depends on openra_rl_training + openra_train.
• eval_core.py: wrap play_episodes_async, force Rust pool.
• agent.py: real OpenAI-compatible model agent w/ minimap PNG.
• Rewrite evaluate.py → fixed-scenario eval producing eval_stats.json.
• compare.py → 95% CI tables. Wire results into app.py leaderboard.
• Validate on rush-hour + scout-maginot (these are the perception tasks).

Phase 1 — Adversarial 1v1 (combat-only, current mechanics)

• Rust: add a second RL-controlled player slot in a 1v1 map (both sides accept Commands; remove scripted-enemy assumption).
• Bench pairwise.py: two-model orchestration, win-rate + Elo, leaderboard.

Phase 2 — Controlled scenario library (current mechanics)

• Author perception/spatial scenarios that isolate the unexplored→target connection failure; maze/chokepoint pathfinding; multi-target prioritization.
• Verifiable rubrics per scenario (intelligence_pct, path-optimality, etc.).

Phase 3 — Rust mechanics expansion (unlocks scenario families)

• 3a Economy: ore/cash/harvester obs + HARVEST cmd + economy reward.
• 3b Production/tech: production queue, BUILD/TRAIN, available_production, power.
• Each sub-phase ships its scenario family + rubrics.

Test coverage (live engine, no mocks)

tests/test_rust_integration.py — 17 tests, ~1.9s, boots real openra_train with rule-based bots (idle / charge / hunter):

• tool correctness: move_units reaches target; idle units hold; attack path; reset schema; same-seed determinism (bit-for-bit).
• corner cases: empty command list, invalid unit id (warns, no raise), invalid attack target, out-of-bounds move — all safe.
• invariants: explored% and units_killed monotonic non-decreasing; discovery set cumulative.
• stack: adapter signal tracking; win-condition predicates + composites
  • unknown-key rejection; deterministic win/fail plumbing (trivially true win/fail conditions ⇒ exact outcome, not bot-skill dependent); all authored packs run end-to-end.

Sequencing (locked)

Phase 0 → Phase 2 (+ P/R/A diagnostics) → Phase 1 → Phase 3

Scenario breadth + per-link diagnostics first: fastest path to exposing real model strengths/weaknesses on current Rust mechanics. Adversarial 1v1 and the economy/production engine work follow.

Model provider abstraction (Phase 0)

openra_bench/agent.py exposes a provider-agnostic agent. Adapters:

• openai-compatible (default): covers local vLLM (matches Training's rollout path) and OpenRouter (test target). Same Chat Completions + multimodal image_url for the minimap PNG. Base URL + key from config/env.
• bedrock: separate adapter (AWS SDK / Converse API), added when needed.

Selected via Bench config (provider, base_url, model, api_key_env). Phase 0 validates with OpenRouter.