Spaces:
Running
Running
| # OpenRA-Bench — Scenario Audit & Gap Analysis | |
| Research + audit. No code changes. Prepared 2026-05-17. | |
| --- | |
| ## Executive Summary | |
| OpenRA-Bench currently ships **85 scenario files** (17 hand-authored | |
| scenario families + 68 procedurally generated `cat-*` packs across 12 | |
| categories, ~200 difficulty levels + 1 TEMPLATE). The design is | |
| unusually well-grounded: every win condition is a verifiable predicate, | |
| difficulty scales by *decision hardness* (info down, decoys up, clock | |
| down, attrition cap) rather than raw numbers, and each pack carries a | |
| `real_world_meaning` + `robotics_analogue`. The anti-memorization | |
| discipline (procedural variation, held-out seeds, generalization-gap | |
| metric) is consistent with current best practice (Procgen, SMACv2, | |
| ARC-AGI). The "RTS reasoning transfers" hypothesis is *directly | |
| supported by published evidence*: lmgame-Bench (arXiv:2505.15146) | |
| showed RL on Sokoban/Tetris lifted Blocksworld (+≥10pts planning) and | |
| WebShop (+6pts multi-turn decisions) but **not** math/coding — i.e. | |
| transfer is real but axis-specific (spatial / planning / embodied), | |
| exactly the axes this suite targets. | |
| **The core finding of this audit:** the capability *taxonomy* is | |
| sound and the transfer story is defensible, but the **coverage is | |
| badly skewed**. Frontier models in 2026 are differentiated almost | |
| entirely on (a) agentic **tool-use / function-calling fidelity** under | |
| strict APIs (BFCL V4, τ²-bench — GPT-5.5 reports 98% τ²-Telecom, | |
| Kimi-K2.6 50% Toolathlon), (b) **long-horizon multi-step agentic | |
| execution** (Terminal-Bench 2.0 — GPT-5.5 82.7%, SWE-bench Verified | |
| ~89%), and (c) **fluid/abstract + spatial reasoning** (ARC-AGI-2 — the | |
| single largest 2026 model delta; ERQA for embodied/spatial). OpenRA-Bench | |
| is *strong* on spatial perception/exploration (FRONT/PERC — parallels | |
| ERQA, the validated transfer target) and *adequate* on constrained | |
| planning (PLAN/TECH/ECON — parallels PlanBench/Blocksworld). It is | |
| **weak on adversarial/game-theoretic reasoning** (no opponent-modeling | |
| scenario despite an RTS engine — the one capability an RTS bench is | |
| uniquely positioned to own), **weak on long-horizon credit assignment** | |
| (most clocks are short single-phase tasks, not 30+ step chains like | |
| Terminal-Bench/OSWorld), and **structurally weak on the very thing | |
| 2026 leaderboards weight most: tool-use/instruction-following fidelity | |
| under a strict action API** — this is *measured* (action-validity | |
| sub-score) but no scenario *isolates* it as the primary objective. | |
| Additionally, ~68 of 85 packs are the auto-generated `cat-*` family: | |
| 12 categories × ~3 levels × ~variants. Many `cat-c5/c6/c7/c8` | |
| "budget-allocation / base-placement" packs are near-duplicates of each | |
| other and of the hand-authored `economy-*` and `building-and-planning` | |
| scenarios — high level count, low *distinct-capability* count. Several | |
| economy packs degenerate to identical tests because harvest income is | |
| 0 on the only loadable map (documented engine-prereq S0/S1) — these | |
| are honest but currently non-discriminating. | |
| **Top recommendations (detail in §4):** | |
| 1. **Add an adversarial/opponent-modeling family** (the unique RTS | |
| value prop; parallels game-theoretic + StarCraft-II-Arena evals). | |
| Activate the Phase-1 1v1 path — this is the highest-leverage gap. | |
| 2. **Add a strict-action-API instruction-following family** isolating | |
| tool-call fidelity (parallels BFCL V4 / τ²-bench — the most | |
| leaderboard-weighted 2026 capability). | |
| 3. **Add 2–3 genuinely long-horizon scenarios** (multi-phase, | |
| 40k+ ticks, credit assignment across opening→tech→assault) to | |
| parallel Terminal-Bench/OSWorld long-task tracks. | |
| 4. **Cut/merge the redundant `cat-*` over-generation** to ~3 distinct | |
| levels per *distinct* decision; the level *count* is inflating | |
| apparent coverage without adding capability breadth. | |
| 5. **Pre-register the transfer panel against ERQA (spatial/embodied), | |
| Blocksworld/PlanBench (planning), and BFCL/τ²-bench (tool-use)** — | |
| ERQA is the empirically-validated correlate for the headline claim; | |
| BFCL validates the under-tested action-fidelity axis. | |
| --- | |
| ## STEP 1 — Scenario Inventory | |
| ### 1.1 Counts | |
| | Group | Files | Notes | | |
| |---|---|---| | |
| | Hand-authored scenario families | 17 | perception/reasoning/action/strategy/economy | | |
| | Generated catalog packs (`cat-*`) | 68 | 12 categories (c1–c12), 5–6 variants × 3 levels each | | |
| | TEMPLATE | 1 | scaffold, not a scenario | | |
| | **Total scenario files** | **85** | ~200 difficulty levels | | |
| The 12 generated categories (from `SCENARIO_CATALOG.md`, verified | |
| against the `cat-*` files): C1 Frontier Scouting (FRONT), C2 Threat | |
| Enumeration (PERC), C3 Tech Critical Path (TECH), C4 Power-Budget | |
| Online (PLAN), C5 Budget Allocation (ECON), C6 Time-Boxed Capital | |
| Deploy (ECON), C7 Defensive-Direction Commit (PERC), C8 Base-Placement | |
| & Staging (PERC), C9 Commit-vs-Retreat (RISK), C10 Force Coordination | |
| (COORD), C11 Tempo/Timing Window (TEMPO), C12 Error Recovery/Replan | |
| (RISK). | |
| ### 1.2 Capability buckets and what each actually tests | |
| The Perception→Reasoning→Action chain is the framing | |
| (`EVAL_STACK_PLAN.md`); every scenario is scored on all three links | |
| with a `weakest_link` diagnostic (`openra_bench/scoring.py`), plus a | |
| continuous `reward_vector` goal tracker (`openra_bench/goal_tracker.py`) | |
| and pass/fail via pure predicates (`openra_bench/scenarios/win_conditions.py`). | |
| | Bucket | Sub-skill | Scenarios | What it actually tests (predicate grammar) | | |
| |---|---|---|---| | |
| | **Perception** | spatial state-estimation under fog | `perception-frontier-reading`, `perception-target-vs-fog`, C2, C7, C8 | `explored_pct_gte`, `enemies/buildings_discovered_gte` under a clock + attrition cap. Read the minimap, steer sensing. | | |
| | **Perception/Front** | which-unexplored-region-to-commit | `reasoning-frontier-commit`, C1 | Frontier selection (pathfinding solved; *choosing* the region is the test). Strong ERQA/Active-Neural-SLAM analogue. | | |
| | **Reasoning** | constrained sequential planning | C3 (tech path), C4 (power budget), `building-and-planning` | Precondition-ordered build to a deadline with `power_surplus_gte≥0`. Blocksworld/PlanBench analogue. | | |
| | **Reasoning/Econ** | resource allocation / multi-objective | `economy-investment`, `economy-time-box`, `economy-force-buildup`, C5, C6, `economy-harvest-*` | Convert a fixed budget into units AND infrastructure under a clock; commit a wide-vs-deep allocation. Harvest variants currently non-discriminating (income=0, see §1.3). | | |
| | **Reasoning/Risk** | risk call / replan under partial info | `reasoning-risk-route`, `strategy-dilemma`, C9 (commit-vs-retreat), C12 (error recovery) | Safe-long vs short-lethal route; engage-vs-hold with attrition cap; rebuild after a setback (`after_ticks` gate). | | |
| | **Action** | multi-unit coordination / execution | `action-multiunit-coordination`, `strategy-twobody`, C10 | Drive dispersed squads to converge in a region with `units_lost_lte:0`. Watch-And-Help / SMAC analogue. | | |
| | **Action** | sequenced execution to deadline | `action-sequenced-execution`, `rush-hour`, `strategy-gauntlet` | Ordered build→deploy→reposition without dropping a step; multi-squad sweep-and-clear. | | |
| | **Tempo** | timing-window discipline | C11 | `after_ticks:t0` then `units_killed_gte:N` within `T1` — act in a window, not before. | | |
| All run on the single Rust-loadable map `rush-hour-arena`. | |
| ### 1.3 Honest limitations already documented in-repo | |
| - **Harvest economy is non-functional** on the only loadable map: the | |
| Python schema rejects `mine`/`gmine`, the `.oramap` files seed no | |
| ore, and `silo` storage is hardcoded inert | |
| (`SCENARIO_BRAINSTORM.md` §"Verified ground-truth constraints"). | |
| Consequence: `economy-harvest-timebox` / `economy-harvest-investment` | |
| / harvest variants degenerate to spend-only tests identical to the | |
| non-harvest economy packs. They are honestly tagged but **currently | |
| non-discriminating** — they inflate the economy-bucket count without | |
| adding a distinct capability. Blocked on engine-prereqs S0 (ore | |
| source) + S1 (silo storage). | |
| - **No enemy-destruction predicate** — offensive success is proxied by | |
| `units_killed_gte` + positional `reach_region`/`building_in_region`. | |
| - **No adversarial slot today** — the 1v1/Elo path is Phase-1, not | |
| built (`EVAL_STACK_PLAN.md`). The "Elo leaderboard" is currently | |
| only meaningful for the fixed-scenario composite, not head-to-head. | |
| --- | |
| ## STEP 2 — What Frontier Benchmarks Measure Today (2026) | |
| Verified via the four required sources + targeted search. URLs in §5. | |
| ### 2.1 Required model pages | |
| - **Qwen3.6-35B-A3B** (HF page, released ~Apr 2026). Reports a | |
| coding/agentic-heavy battery: SWE-bench Verified 73.4, SWE-bench | |
| Pro 49.5, Terminal-Bench 2.0 51.5, **TAU3-Bench 67.2**, Tool | |
| Decathlon 26.9, MCPMark 37.0; reasoning: MMLU-Pro 85.2, GPQA 86.0, | |
| AIME26 92.7; **spatial-intelligence block: EmbSpatialBench 84.3, | |
| RefSpatialBench 64.3, RefCOCO 92.0**. Takeaway: even a mid-size | |
| model now reports a *dedicated embodied-spatial benchmark block* and | |
| a *dedicated tool/agent block* — these are the differentiating axes. | |
| - **Kimi-K2.6** (HF page). Agentic-first: OSWorld-Verified 73.1, | |
| **Toolathlon 50.0, MCPMark 55.9**, BrowseComp 83.2, Terminal-Bench | |
| 2.0 66.7, SWE-bench Verified 80.2; reasoning AIME 2026 96.4, | |
| GPQA-Diamond 90.5. Tool-use / computer-use / long-horizon browsing | |
| dominate the card. | |
| - **GPT-5.5** (OpenAI announcement; page is 403 to direct fetch, | |
| corroborated via search). Released ~Apr 23 2026. Headline numbers: | |
| **SWE-bench Verified 88.7**, SWE-bench Pro 58.6, **Terminal-Bench | |
| 2.0 82.7 (SOTA)**, **τ²-bench Telecom 98.0**, **ARC-AGI-2 85.0 | |
| (+11.7 over prior, the launch's single largest delta)**, GDPval | |
| 84.9. The two most-emphasized capabilities: agentic | |
| coding/tool-orchestration and ARC-AGI-2 fluid reasoning. | |
| - **Artificial Analysis** leaderboard. The page no longer exposes | |
| components inline, but the current **Intelligence Index v4.0** = | |
| 10 evals: **GDPval-AA, τ²-Bench Telecom, Terminal-Bench Hard, | |
| SciCode, AA-LCR (long-context reasoning), AA-Omniscience, IFBench | |
| (instruction-following), Humanity's Last Exam, GPQA Diamond, | |
| CritPt**. Note the heavy weighting toward *agentic tool-use* | |
| (τ²-bench), *long-horizon terminal agency* (Terminal-Bench Hard), | |
| and *instruction-following fidelity* (IFBench). Pass@1, ±<1% CI. | |
| ### 2.2 Broadly-used benchmarks, the capability each isolates, and the | |
| real-world proxy | |
| | Benchmark | Isolates | Real-world proxy | How models differentiate (2026) | | |
| |---|---|---|---| | |
| | **SWE-bench Verified / Pro** | repo-level multi-file code editing to a passing test | autonomous coding agents | Top cluster ~80–89% Verified (GPT-5.5 88.7); Pro still <60% — the hard frontier | | |
| | **Terminal-Bench 2.0** | long-horizon shell agency: plan→iterate→tool-coordinate, deterministic check | CLI/devops automation agents | Wide spread: GPT-5.5 82.7 vs Qwen3.6 51.5 — strong separator | | |
| | **τ-bench / τ²-bench (Telecom)** | multi-turn **tool use under a strict policy/API**, dialog state | customer-service / API agents | Near-saturating at the top (GPT-5.5 98.0) but mid-tier collapses — fidelity cliff | | |
| | **BFCL V4** | function-calling: parallel/multiple selection, **relevance (when NOT to call)**, multi-turn, agentic memory | tool/function-calling fidelity | AST-graded; single-turn solved, multi-turn/relevance still discriminating | | |
| | **GAIA** | compound multi-step tool-use + reasoning across web/files | general assistant agents | Multi-hop chains expose planning/credit-assignment failures | | |
| | **OSWorld** | full GUI computer control, 15- vs 50-step tasks | computer-use agents | Long-task tracks isolate horizon length specifically | | |
| | **ARC-AGI-2** | fluid abstract pattern induction, anti-memorization | genuine generalization | Largest 2026 deltas; GPT-5.5 85.0 — the prestige reasoning metric | | |
| | **GPQA / AIME / MMLU-Pro / HLE** | hard knowledge + math reasoning | expert QA | Largely saturating top-end except HLE — *not* a transfer target for game RL (lmgame-Bench) | | |
| | **ERQA** | **embodied spatial / trajectory / state-estimation / multi-view / task reasoning** (400 MCQ VQA) | robotics perception+planning | CoT moves it only +4–6.5pts → *reasoning-shaping* finetune is what moves it; the validated OpenRA-Bench transfer target | | |
| | **PlanBench / Blocksworld** | precondition-ordered sequential planning, cost-optimal, replanning | task-graph scheduling, robot task planning | Mystery-Blocksworld defeats memorization; *the* planning correlate | | |
| | **lmgame-Bench (Sokoban/Tetris RL → transfer)** | game-RL → spatial/planning transfer | the OpenRA-Bench hypothesis itself | RL→Blocksworld +≥10, →WebShop +6; NOT →GSM8K/coding | | |
| | **SMAC → SMACv2** | multi-agent micro; the anti-memorization cautionary tale | strategy-game RL | Open-loop policy beat SMAC ⇒ procedural variation mandatory (the suite follows this) | | |
| Net 2026 picture: differentiation has moved off knowledge/math QA | |
| (saturating) onto **agentic tool-use fidelity, long-horizon execution, | |
| computer/embodied control, and fluid reasoning (ARC-AGI-2)**. | |
| --- | |
| ## STEP 3 — Capability ↔ Coverage ↔ Parallel Benchmark Gap Table | |
| | Capability / real-world use case | OpenRA-Bench coverage | Parallel benchmark | Verdict | | |
| |---|---|---|---| | |
| | Spatial state-estimation / perception under partial obs | `perception-target-vs-fog`, `perception-frontier-reading`, C2, C7, C8 | **ERQA** (spatial/state-est), SpatialVLM, VSR | **Well-covered** — and this is the empirically-validated transfer target | | |
| | Frontier exploration: which unknown region to commit | `reasoning-frontier-commit`, C1 | Active Neural SLAM, frontier exploration, ERQA-trajectory | **Well-covered** | | |
| | Constrained sequential planning (precondition graph + deadline) | C3, C4, `building-and-planning` | **PlanBench / Blocksworld**, ALFWorld | **Adequate** (parallels exist; single-map limits variety) | | |
| | Resource allocation / multi-objective under a budget | `economy-investment`, `economy-time-box`, C5, C6 | PlanBench cost-optimal, SmartPlay | **Adequate** (spend-only) — harvest variants **non-discriminating** (engine S0/S1) | | |
| | Risk assessment / replanning under partial info | `reasoning-risk-route`, `strategy-dilemma`, C9, C12 | PlanBench replanning, StarCraft-II-Arena | **Adequate** | | |
| | Long-horizon multi-step credit assignment | thin: most clocks are short single-phase; C12 only gates with `after_ticks` | **Terminal-Bench 2.0**, OSWorld 50-step, GAIA | **Weak** — no genuine opening→tech→assault chain | | |
| | Multi-unit / multi-agent coordination | `action-multiunit-coordination`, `strategy-twobody`, C10 | Watch-And-Help, SMAC(v2) | **Adequate** | | |
| | Tool-use / function-calling **fidelity under a strict action API** | only as a *cross-cutting score* (`actions_warned/issued`); no scenario isolates it as the objective | **BFCL V4**, **τ²-bench**, IFBench | **Weak/structural gap** — yet this is the most leaderboard-weighted 2026 axis | | |
| | Instruction-following under strict constraints | implicit in win predicates; not isolated | **IFBench** (in AA Index v4) | **Weak** | | |
| | Adversarial / game-theoretic reasoning (opponent modeling, deception, counter-strategy) | **none** — no live adversary; `rush-hour` enemy is scripted; 1v1/Elo is Phase-1 unbuilt | StarCraft-II-Arena, game-theory evals; *the unique RTS value prop* | **Missing** — largest strategic gap | | |
| | Tempo / timing-window discipline | C11 | TextStarCraft II, SmartPlay | **Adequate** (but only 1 category, 15 levels) | | |
| | Fluid/abstract anti-memorization generalization | enforced by procedural variation + held-out seeds + generalization-gap metric | **ARC-AGI-2** philosophy | **Methodologically well-covered** (design discipline is correct) | | |
| ### Scenarios with weak / no transfer story (flag) | |
| - **`rush-hour` / `strategy-gauntlet` "sweep-and-clear"** — primarily a | |
| search-and-destroy execution task. Defensible as multi-robot | |
| patrol-and-clear, but with a *scripted* enemy it tests execution, | |
| not adversarial reasoning. Keep, but it should not be load-bearing | |
| for the "strategy" claim. | |
| - **`economy-harvest-timebox` / `economy-harvest-investment`** — | |
| currently economically identical to the non-harvest economy packs | |
| (income=0). Honest, but they are *placeholders*, not distinct tests; | |
| do not count them as economy coverage until S0/S1 land. | |
| - **`cat-c5/c6/c7/c8` over-generation** — many variants differ only in | |
| region coordinates / target counts. The *decision* is the same; this | |
| is level inflation, not capability breadth (mild | |
| SMAC-style memorization risk if seeds aren't truly held out per | |
| variant). Not "game trivia," but low marginal information. | |
| - The suite has **no** scenario that is pure game-trivia with zero | |
| transfer story — the `real_world_meaning`/`robotics_analogue` | |
| discipline is doing its job. The problem is *skew and redundancy*, | |
| not arbitrariness. | |
| --- | |
| ## STEP 4 — Concrete, Prioritized Edit Advice | |
| ### P0 — Add an adversarial / opponent-modeling family (highest leverage) | |
| An RTS engine's unique, defensible value vs every other benchmark is | |
| **live adversarial / game-theoretic reasoning** — and the suite | |
| currently has *zero*. Build the Phase-1 second RL-controlled slot | |
| (`EVAL_STACK_PLAN.md` Phase 1) and add: | |
| - **Counter-Strategy Read**: opponent commits an observable opening | |
| (rush vs tech vs expand); agent must read it from partial recon and | |
| pick the dominant counter. Capability: adversarial reasoning + | |
| perception. Real-world: competitive multi-agent / negotiation / | |
| red-teaming. Parallel: **StarCraft-II-Arena (ICLR'25)**, | |
| game-theoretic LLM evals. | |
| - **Deception / feint handling**: opponent shows a decoy force; win | |
| predicate rewards committing against the *real* axis | |
| (`building_in_region`/`units_killed_gte` keyed off the true threat). | |
| Parallel: adversarial robustness, opponent modeling. | |
| This also makes the **Elo leaderboard** genuinely meaningful | |
| (head-to-head), which it currently is not. | |
| ### P0 — Add a strict-action-API instruction-following family | |
| The most leaderboard-weighted 2026 axis (τ²-bench in AA Index v4, | |
| BFCL V4, IFBench) is only measured as a *side diagnostic* here. Add a | |
| family whose **primary objective is action-API fidelity**: | |
| - Tasks where the win predicate is only reachable via a *specific | |
| command sequence/format under explicit policy constraints* (e.g. | |
| "achieve X but never issue `attack` before tick T", "use only | |
| `move_units` + `deploy`, no `build`"), scoring relevance (issuing a | |
| disallowed call = fail, à la BFCL "relevance detection"). | |
| - Capability: instruction-following + tool-call fidelity under a | |
| strict API. Real-world: agentic API/tool orchestration, policy | |
| compliance. Parallel: **BFCL V4 (relevance/multi-turn), τ²-bench, | |
| IFBench**. | |
| ### P1 — Add 2–3 genuinely long-horizon, multi-phase scenarios | |
| Current clocks are mostly short single-phase. Add scenarios spanning | |
| opening→scout→economy→tech→army→engagement in **one episode (40k+ | |
| ticks)** with a terminal objective, so the score depends on **credit | |
| assignment across phases** (an early scouting error must surface as a | |
| late failure). Capability: long-horizon credit assignment. Parallel: | |
| **Terminal-Bench 2.0, OSWorld 50-step track, GAIA** multi-hop. This is | |
| where 2026 models separate most and the suite is currently thin. | |
| ### P1 — Cut / merge the redundant `cat-*` over-generation | |
| Reduce each generated category to **≤3 levels per genuinely distinct | |
| decision** instead of 5–6 near-duplicate coordinate variants. Net | |
| ~200→~90 levels but *higher distinct-capability density* and lower | |
| memorization risk. Specifically merge: C5↔C6 (both "budget→units+ | |
| buildings" ECON), C7↔C8 (both "place building in inferred region" | |
| PERC). Keep the procedural seed variation *within* a level (the | |
| anti-memorization mechanism) — cut the redundant *level* multiplication. | |
| ### P1 — Quarantine non-discriminating economy packs | |
| Tag `economy-harvest-*` (and any harvest variant) as **engine-prereq / | |
| not-scored** until S0 (ore source) + S1 (silo storage) land | |
| (`SCENARIO_BRAINSTORM.md`). Do not count them in economy coverage | |
| claims. Land S0 (a one-line `VALID_ACTOR_TYPES` add or a | |
| `resource_fields` scenario field) — it is cheap and unblocks a real, | |
| distinct economy-throughput capability. | |
| ### Over/under-representation summary | |
| - **Over-represented:** ECON spend-allocation (C5/C6 + 5 hand-authored | |
| economy packs, several non-discriminating), and generated PERC | |
| region-placement (C7/C8). Level count >> capability count. | |
| - **Under-represented:** adversarial/game-theoretic (zero), | |
| tool-use/API-fidelity isolation (zero dedicated), long-horizon | |
| credit assignment (thin), tempo (1 category). | |
| - **Right-sized:** spatial perception/frontier (the validated transfer | |
| target — keep strong), coordination, risk/replan. | |
| ### Strengthening the generalization-transfer argument | |
| The headline claim ("rush-hour finetune lifted ERQA") needs a | |
| pre-registered external transfer panel, scored as **per-axis deltas, | |
| not aggregate** (lmgame-Bench protocol): | |
| 1. **ERQA** — the *primary* correlate. It is reasoning-sensitive | |
| (CoT-only moves it just +4–6.5pts), embodied/spatial, and is the | |
| axis the observed transfer hit. Report ERQA spatial / trajectory / | |
| task subscores separately. Strongest evidence for the claim. | |
| 2. **Blocksworld / PlanBench (incl. Mystery-Blocksworld)** — | |
| validates the PLAN/TECH/ECON families; this is the exact transfer | |
| lmgame-Bench demonstrated (RL→Blocksworld +≥10pts) and defeats | |
| memorization. | |
| 3. **BFCL V4 / τ²-bench** — validates the *new* P0 action-fidelity | |
| family and tests whether strict-action-API discipline transfers | |
| (the most commercially relevant axis; currently untested here). | |
| 4. **Negative controls: GSM8K + a coding eval (e.g. LiveCodeBench)** — | |
| lmgame-Bench showed game-RL does **not** transfer to math/coding; | |
| include these to demonstrate the transfer is *specific* (spatial/ | |
| planning), not a generic capability bump. This negative result is | |
| what makes the positive claim credible. | |
| 5. Continue reporting the **generalization gap on held-out seeds** | |
| (Procgen/SMACv2/ARC-AGI discipline) — already designed in; keep it | |
| front-and-center as the anti-memorization guarantee. | |
| --- | |
| ## STEP 5 — Sources (verified via search/fetch, 2026-05) | |
| - Qwen3.6-35B-A3B model card — https://huggingface.co/Qwen/Qwen3.6-35B-A3B | |
| - Kimi-K2.6 model card — https://huggingface.co/moonshotai/Kimi-K2.6 | |
| - Introducing GPT-5.5 — https://openai.com/index/introducing-gpt-5-5/ (direct fetch 403; corroborated via search: interestingengineering.com/ai-robotics/opanai-gpt-5-5-agentic-coding-gains, kingy.ai GPT-5.5 benchmarks, llm-stats.com/models/gpt-5.5) | |
| - Artificial Analysis leaderboard — https://artificialanalysis.ai/leaderboards/models ; methodology — https://artificialanalysis.ai/methodology/intelligence-benchmarking ; Intelligence Index — https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index | |
| - BFCL V4 — https://gorilla.cs.berkeley.edu/leaderboard.html ; paper https://openreview.net/forum?id=2GmDdhBdDk | |
| - τ²-bench (Sierra) via Agentic AI Benchmarks — https://awesomeagents.ai/leaderboards/agentic-ai-benchmarks-leaderboard/ | |
| - ERQA — https://github.com/embodiedreasoning/ERQA ; Gemini Robotics arXiv:2503.20020 https://arxiv.org/html/2503.20020v1 | |
| - lmgame-Bench arXiv:2505.15146 — https://arxiv.org/abs/2505.15146 | |
| - GAIA / OSWorld / Terminal-Bench / ARC-AGI-2 overview — https://www.marktechpost.com/2026/04/26/top-7-benchmarks-that-actually-matter-for-agentic-reasoning-in-large-language-models/ ; https://www.spheron.network/blog/ai-agent-benchmarking-gpu-cloud-swebench-gaia/ | |
| - In-repo: `SCENARIO_CATALOG.md`, `SCENARIO_BRAINSTORM.md`, `EVAL_STACK_PLAN.md`, `openra_bench/scenarios/win_conditions.py`, `openra_bench/scoring.py`, `openra_bench/goal_tracker.py`, `openra_bench/scenarios/packs/*.yaml` | |
| - Supporting literature cited in-repo and corroborated: SMAC→SMACv2 (arXiv:2212.07489), Procgen (arXiv:1912.01588), ARC-AGI (arXiv:1911.01547), PlanBench (arXiv:2206.10498), Active Neural SLAM (arXiv:2004.05155) | |