Merge pull request #5 from yiyu-tian/scenarios/tech-production-planning

SCENARIO_BRAINSTORM.md

@@ -341,3 +341,74 @@ full (b) **needs S0 + S1**.
 - `openra_bench/scenarios/packs/economy-time-box.yaml` — validator: `OK`
 - `openra_bench/scenarios/packs/economy-investment.yaml` — validator: `OK`
 - `SCENARIO_BRAINSTORM.md` (this file)
+
+---
+
+## Tech-Tree & Production Planning pack (design-doc Family 2)
+
+Pack: `openra_bench/scenarios/packs/tech-production-planning.yaml`. Maps
+the Benchmark Design Proposal's **Task Family 2 — Sequential Dependency
+Planning** (Scenarios 2A/2B/2C) directly into one bench pack.
+
+**Why a new pack rather than extending existing ones.** Two existing
+packs touch this territory but ladder on different axes:
+
+| Pack | Easy axis | Medium axis | Hard axis | Ladder shape |
+|---|---|---|---|---|
+| `building-and-planning` | 1-hop prereq (powr→tent) + building count | spatial direction (place pbox east) | MCV relocate / new base | building → spatial → mobility (changes axis per level) |
+| `strict-production-bom` | exact 3×e1 (no overproduction) | + 2×e3 | + tsla + tight budget | spec-fidelity (BFCL/τ²-bench style; orthogonal to dependency planning) |
+| **`tech-production-planning`** (new) | 2-hop chain `weap→fix→3tnk` | parallel branches (barr ‖ weap+fix) + `power_surplus_gte:0` forces 2nd powr | mid-game infantry tech committed → scout fog → pivot to anti-armor | **dependency depth → parallel breadth → adaptive replanning** (same axis at increasing complexity) |
+
+`building-and-planning` is mixed-axis (Family 2-ish at easy, drifts into
+Family 3/spatial later). `strict-production-bom` tests a different
+capability entirely (instruction-following fidelity, not dependency
+planning). The new pack is the first time Family 2 is ladded canonically
+on one axis end-to-end.
+
+**Real tech tree (engine ground truth from `openra_env/game_data.py`):**
+
+| Building | Prereqs | Power |
+|---|---|---|
+| `powr` | — | +100 |
+| `proc` (refinery) | `powr` | −30 |
+| `weap` (war factory) | `proc` | −30 |
+| `fix` (service depot) | `weap` | −30 |
+| `barr` (soviet barracks) | `powr` | −20 |
+| `3tnk` (Heavy Tank) | `weap` + `fix` | unit |
+| `e1` / `e3` (infantry) | `barr|tent` | unit |
+
+→ Heavy Tank's real chain is `powr → proc → weap → fix → 3tnk` (4
+prereqs deep). Building all of `proc + weap + fix + barr` drains to −10
+power against a 100-power base — a SECOND powr is mandatory for medium.
+This is the real `power_surplus_gte:0` teeth at medium.
+
+**Difficulty calibration (one notch under doc spec).** Design doc 2A
+("Construction Yard + 5000 credits → produce one Heavy Tank from
+scratch") is harder than every existing easy in the repo (~3× the
+work). Easy here pre-places `fact + powr + proc` so the chain becomes
+`weap → fix → 3tnk` (2-hop), aligning with `strict-production-bom`
+easy's "1 build + N train" workload. Medium and hard are calibrated
+similarly downward. Hard (adaptive transition under fog) is left at the
+design-doc difficulty because no existing pack tests online plan
+revision — this is the pack's core new contribution.
+
+### Follow-ups (Design Proposal Families 4 / 5 / 7) — separate PRs
+
+Each likely needs new engine infrastructure that isn't yet wired into
+the bench:
+
+- **Family 4 — Adversarial Reasoning** (Beginner/Normal/Hard AI
+  match). Existing `adversarial-*.yaml` packs are duels/sieges (combat
+  encounters), not full RTS matches; `bot_type: ''` is empty across
+  them, so a scripted opponent AI tier doesn't appear to be exposed
+  yet. Needs `bot_type` integration + a "win full match" win predicate.
+
+- **Family 5 — Reactive Replanning** (Early threat / Multi-raid /
+  Catastrophic). The `interrupts` field exists (`adversarial-duel`
+  uses `enemy_unit_spotted` + `own_unit_destroyed`), but
+  raid-generation hooks (mid-game enemy attack scripts) and an
+  "infrastructure destroyed" trigger aren't represented in any pack.
+
+- **Family 7 — Holistic Strategic Intelligence** (full RTS match at 3
+  difficulties). Needs Family 4's infra (full-match predicate + AI
+  opponent) plus long-horizon scoring beyond the existing rubric.

openra_bench/scenarios/packs/tech-production-planning.yaml

@@ -0,0 +1,127 @@
+meta:
+  id: tech-production-planning
+  title: 'Tech-Tree & Production — Sequential Dependency Planning'
+  capability: reasoning
+  real_world_meaning: >
+    Sequential dependency planning under hard precedence + power + budget
+    constraints. Easy is a 2-hop prereq chain (build weap → fix → train
+    one Heavy Tank). Medium forces parallel branches (infantry queue at
+    barr AND vehicle queue at weap+fix) under a power brownout that
+    requires a second power plant. Hard requires online plan revision:
+    the agent starts mid-game with infantry tech committed, must scout
+    through fog to reveal that the enemy is heavy armor, and pivot
+    production to anti-armor (Rocket Soldier) before the deadline. The
+    decision being tested is build-order scheduling under precedence,
+    not motor control of any single build.
+  robotics_analogue: >
+    Task-graph scheduling with multi-hop precedence and resource
+    constraints, escalating to mid-execution replanning when discovered
+    facts invalidate the initial plan — PlanBench Mystery-Blocksworld
+    (precedence) plus an online-revision twist after partial observation.
+  author: openra-bench
+
+base_map: rush-hour-arena
+
+base:
+  agent:
+    faction: soviet
+    cash: 0
+  enemy:
+    faction: allies
+    cash: 0
+    bot_type: ''
+  tools:
+    - build
+    - place_building
+    - train
+    - cancel_production
+    - move_units
+    - deploy
+    - stop
+    - set_rally_point
+  planning: true
+  termination:
+    max_ticks: 40000
+  actors:
+    - {type: fact, owner: agent, position: [10, 18]}
+    - {type: powr, owner: agent, position: [14, 18]}
+
+levels:
+  easy:
+    description: >
+      Simple Tech Chain — fact, powr, and proc are pre-placed; build
+      War Factory → Service Depot, then train one Heavy Tank. The
+      decision is recognising 3tnk needs weap + fix in that order; the
+      budget covers the chain with modest slack. Tests the agent can
+      sequence a deep enough prereq chain (proc→weap→fix→3tnk) without
+      attempting to train 3tnk before fix exists.
+    starting_cash: 5000
+    overrides:
+      actors:
+        - {type: fact, owner: agent, position: [10, 18]}
+        - {type: powr, owner: agent, position: [14, 18]}
+        - {type: proc, owner: agent, position: [16, 22]}
+        - {type: mine, owner: agent, position: [20, 24]}
+        - {type: e1, owner: enemy, position: [114, 34], stance: 2}
+    win_condition:
+      all_of:
+        - unit_type_count_gte: {type: 3tnk, n: 1}
+        - building_count_gte: {type: fix, n: 1}
+        - within_ticks: 18000
+    max_turns: 60
+
+  medium:
+    description: >
+      Parallel Dependency Optimization — only fact + powr pre-placed.
+      Produce one Heavy Tank PLUS one Rocket Soldier PLUS three Rifle
+      Infantry, with power_surplus ≥ 0 at the deadline. The combined
+      build (proc + barr + weap + fix = -110 power) drains net power
+      below zero against a 100-power base, so a SECOND power plant is
+      mandatory. Tight clock forces queueing the infantry branch (barr)
+      in parallel with the vehicle branch (weap+fix); serial production
+      blows the clock on idle time.
+    starting_cash: 6500
+    overrides:
+      actors:
+        - {type: fact, owner: agent, position: [10, 18]}
+        - {type: powr, owner: agent, position: [14, 18]}
+        - {type: mine, owner: agent, position: [20, 24]}
+        - {type: e1, owner: enemy, position: [114, 34], stance: 2}
+    win_condition:
+      all_of:
+        - unit_type_count_gte: {type: 3tnk, n: 1}
+        - unit_type_count_gte: {type: e3, n: 1}
+        - unit_type_count_gte: {type: e1, n: 3}
+        - power_surplus_gte: 0
+        - within_ticks: 22000
+    max_turns: 80
+
+  hard:
+    description: >
+      Adaptive Tech Transition — mid-game start: fact + powr + proc +
+      barr and three Rifle Infantry are already on the field (an
+      infantry build committed). Three enemy Heavy Tanks sit passive on
+      the east side, hidden by fog. The agent must scout, recognise
+      that infantry is the wrong counter to heavy armor, and pivot
+      production to FIVE Rocket Soldiers (e3, anti-armor) before the
+      deadline. Overproducing infantry past ten = fail (doubling down
+      on the wrong plan exhausts the budget). The transition — cancel
+      planned infantry, fund anti-armor, scout — is the test.
+    starting_cash: 3000
+    overrides:
+      actors:
+        - {type: fact, owner: agent, position: [10, 18]}
+        - {type: powr, owner: agent, position: [14, 18]}
+        - {type: proc, owner: agent, position: [16, 22]}
+        - {type: barr, owner: agent, position: [12, 21]}
+        - {type: e1, owner: agent, position: [8, 16], count: 3}
+        - {type: mine, owner: agent, position: [20, 24]}
+        - {type: 3tnk, owner: enemy, position: [110, 34], stance: 2, count: 3}
+    win_condition:
+      all_of:
+        - enemies_discovered_gte: 2
+        - unit_type_count_gte: {type: e3, n: 5}
+        - within_ticks: 14000
+    fail_condition:
+      unit_type_count_gte: {type: e1, n: 10}
+    max_turns: 90

tests/test_tech_production_planning.py

@@ -0,0 +1,131 @@
+"""tech-production-planning pack — full loop on Rust.
+
+Canonical Family 2 (Sequential Dependency Planning) from the Benchmark
+Design Proposal: easy = 2-hop chain (weap -> fix -> 3tnk), medium =
+parallel branches under power_surplus_gte:0, hard = adaptive transition.
+
+A scripted "tech-planner" agent walks the easy chain (build War Factory
+-> Service Depot -> train one Heavy Tank) and proves the declarative
+win_condition fires; medium/hard run deterministically with the same
+agent (they need richer play than a scripted depth-first chain to win,
+so we assert determinism + budget bound, mirroring test_building_planning).
+"""
+
+from __future__ import annotations
+
+import pytest
+
+pytest.importorskip("openra_train", reason="Rust env wheel not installed")
+
+from openra_bench.eval_core import run_level
+from openra_bench.scenarios import load_pack
+from openra_bench.scenarios.loader import PACKS_DIR, compile_level
+from openra_bench.scoring import score_episode
+
+PACK = PACKS_DIR / "tech-production-planning.yaml"
+
+
+def tech_planner_agent(render_state, Command):
+    """Walk the 3tnk prereq chain depth-first.
+
+    Pre-placed: fact, powr, proc. Build weap -> fix, then train one 3tnk.
+    `build` queues the item; `place_building` no-ops until the building
+    is ready to be placed.
+    """
+    btypes = [b["type"] for b in render_state.get("own_buildings", [])]
+    producing = render_state.get("production", []) or []
+    units = [u["type"] for u in render_state.get("units_summary", [])]
+
+    if "weap" not in btypes:
+        cmds = []
+        if "weap" not in producing:
+            cmds.append(Command.build("weap"))
+        cmds.append(Command.place_building("weap", 18, 22))
+        return cmds
+    if "fix" not in btypes:
+        cmds = []
+        if "fix" not in producing:
+            cmds.append(Command.build("fix"))
+        cmds.append(Command.place_building("fix", 22, 22))
+        return cmds
+    if "3tnk" not in units:
+        if "3tnk" not in producing:
+            return [Command.build("3tnk")]
+        return [Command.observe()]
+    return [Command.observe()]
+
+
+def test_pack_compiles_with_three_levels():
+    pack = load_pack(PACK)
+    assert pack.meta.id == "tech-production-planning"
+    assert pack.meta.capability == "reasoning"
+    assert set(pack.levels) == {"easy", "medium", "hard"}
+
+
+def test_easy_tech_chain_wins_and_scores():
+    pack = load_pack(PACK)
+    c = compile_level(pack, "easy")
+    assert c.map_supported
+    assert c.starting_cash == 5000
+
+    res = run_level(c, tech_planner_agent, seed=1)
+
+    # The chain went through both weap AND fix (the prereq for 3tnk).
+    btypes = res.signals.own_building_types
+    assert "weap" in btypes and "fix" in btypes, (
+        f"chain incomplete; own_building_types={btypes}"
+    )
+    # The agent actually trained a Heavy Tank.
+    unit_types = [u["type"] for u in res.signals.units_summary]
+    assert "3tnk" in unit_types, f"no 3tnk produced; units={unit_types}"
+    # Production debited the budget.
+    assert res.signals.cash < 5000, "construction must debit the budget"
+    assert res.outcome == "win", f"easy should be winnable, got {res.outcome}"
+
+    sc = score_episode(c, res)
+    assert sc.outcome == "win" and 0.0 <= sc.composite <= 1.0
+
+
+def test_easy_run_is_deterministic():
+    c = compile_level(load_pack(PACK), "easy")
+    a = run_level(c, tech_planner_agent, seed=7)
+    b = run_level(c, tech_planner_agent, seed=7)
+    assert (a.outcome, a.turns, a.signals.cash) == (
+        b.outcome,
+        b.turns,
+        b.signals.cash,
+    ), "same seed must be deterministic"
+
+
+@pytest.mark.parametrize("level", ["medium", "hard"])
+def test_medium_hard_run_and_score_within_constraints(level):
+    """Harder levels need richer play than the scripted depth-first chain
+    (medium wants parallel queues + 2nd powr; hard wants scouting +
+    pivot to anti-armor). Assert they execute within declared constraints
+    and score deterministically — the contributor loop must be robust
+    even on a loss."""
+    pack = load_pack(PACK)
+    c = compile_level(pack, level)
+    a = run_level(c, tech_planner_agent, seed=3)
+    b = run_level(c, tech_planner_agent, seed=3)
+    assert a.outcome in {"win", "draw", "loss"}
+    assert (a.outcome, a.turns, a.signals.cash) == (
+        b.outcome,
+        b.turns,
+        b.signals.cash,
+    ), "same seed must yield identical outcome"
+    sc = score_episode(c, a)
+    assert 0.0 <= sc.composite <= 1.0
+    # Budget constraint enforced by the engine.
+    assert a.signals.cash <= c.starting_cash
+
+
+def test_hard_fail_predicate_set():
+    """Hard declares overproducing infantry past 10 = fail. Confirm the
+    fail_condition is wired through compile (the engine actually checks
+    it). Schema-level check, no episode run required."""
+    c = compile_level(load_pack(PACK), "hard")
+    assert c.fail_condition is not None
+    # The fail leaf is unit_type_count_gte on e1 at n=10.
+    fc = c.fail_condition.model_dump(exclude_none=True)
+    assert fc == {"unit_type_count_gte": {"type": "e1", "n": 10}}, fc