| # GridOps — Agent Leaderboard |
|
|
| Benchmark results across 10 agents on the GridOps OpenEnv environment. |
| All runs use seed=42 for full reproducibility. Each task is 72 steps (3 days). |
|
|
| --- |
|
|
| ## Overall Standings |
|
|
| | Rank | Agent | Task 1 (Normal) | Task 2 (Heatwave) | Task 3 (Crisis) | **Average** | |
| |---:|---|:---:|:---:|:---:|:---:| |
| | 1 | **Grok-4 (xAI)** | 0.80 | **0.82** | **0.72** | **0.78** | |
| | 2 | Oracle (rule-based) | 0.79 | 0.81 | 0.70 | 0.77 | |
| | 3 | **GPT-5.4 (OpenAI)** | 0.79 | 0.79 | 0.67 | 0.75 | |
| | 4 | Gemma-4-31B (Google) | **0.81** | 0.79 | 0.62 | 0.74 | |
| | 4 | Grok 4.20 Multi-Agent | **0.81** | 0.80 | 0.60 | 0.74 | |
| | 4 | DeepSeek V3.2 | 0.80 | 0.79 | 0.62 | 0.74 | |
| | 7 | GPT-5.4-mini (OpenAI) | 0.72 | 0.74 | 0.46 | 0.64 | |
| | 8 | Qwen 3.6 Plus (free) | 0.69 | 0.67 | 0.45 | 0.60 | |
| | 9 | Gemini 3.1 Pro Preview | 0.65 | 0.53 | 0.47 | 0.55 | |
| | 10 | Kimi K2.5 | 0.57 | 0.54 | 0.48 | 0.53 | |
| | — | Do-Nothing baseline | 0.58 | 0.51 | 0.45 | 0.51 | |
| | — | Always-Discharge | 0.59 | 0.51 | 0.45 | 0.52 | |
| | — | Always-Diesel | 0.42 | 0.42 | 0.44 | 0.43 | |
|
|
| --- |
|
|
| ## Capability Tiers |
|
|
| | Tier | Score Range | Agents | |
| |---|---|---| |
| | **Frontier** | 0.74 - 0.78 | Grok-4, GPT-5.4, Gemma-4-31B, Grok 4.20, DeepSeek V3.2 | |
| | **Hand-coded baseline** | 0.77 | Oracle (rule-based) | |
| | **Mid-tier** | 0.60 - 0.64 | GPT-5.4-mini, Qwen 3.6 Plus | |
| | **Weak** | 0.51 - 0.55 | Kimi K2.5, Gemini 3.1 Pro Preview | |
| | **No-intelligence baselines** | 0.43 - 0.52 | Do-Nothing, Always-Discharge, Always-Diesel | |
|
|
| --- |
|
|
| ## Per-Task Breakdown |
|
|
| ### Task 1: Normal Summer (Easy) |
| *Tests basic battery arbitrage. ~100 kW avg demand, Rs 3-12 prices, no heatwave.* |
|
|
| | Rank | Agent | Score | |
| |---:|---|:---:| |
| | 1 | Gemma-4-31B | **0.81** | |
| | 1 | Grok 4.20 Multi-Agent | **0.81** | |
| | 3 | Grok-4 | 0.80 | |
| | 3 | DeepSeek V3.2 | 0.80 | |
| | 5 | Oracle | 0.79 | |
| | 5 | GPT-5.4 | 0.79 | |
| | 7 | GPT-5.4-mini | 0.72 | |
| | 8 | Qwen 3.6 Plus | 0.69 | |
| | 9 | Gemini 3.1 Pro Preview | 0.65 | |
| | 10 | Always-Discharge | 0.59 | |
| | 11 | Do-Nothing | 0.58 | |
| | 12 | Kimi K2.5 | 0.57 | |
| | 13 | Always-Diesel | 0.42 | |
|
|
| ### Task 2: Heatwave + Price Spike (Medium) |
| *Tests temporal planning. Day 2-3 heatwave (+30% demand), Rs 20 evening price spike visible in 4h forecast.* |
|
|
| | Rank | Agent | Score | |
| |---:|---|:---:| |
| | 1 | Grok-4 | **0.82** | |
| | 2 | Oracle | 0.81 | |
| | 3 | Grok 4.20 Multi-Agent | 0.80 | |
| | 4 | Gemma-4-31B | 0.79 | |
| | 4 | DeepSeek V3.2 | 0.79 | |
| | 4 | GPT-5.4 | 0.79 | |
| | 7 | GPT-5.4-mini | 0.74 | |
| | 8 | Qwen 3.6 Plus | 0.67 | |
| | 9 | Kimi K2.5 | 0.54 | |
| | 10 | Gemini 3.1 Pro Preview | 0.53 | |
| | 11 | Do-Nothing | 0.51 | |
| | 11 | Always-Discharge | 0.51 | |
| | 13 | Always-Diesel | 0.42 | |
|
|
| ### Task 3: Extreme Crisis + Grid Outage (Hard) |
| *Tests constraint management. Full 3-day heatwave, -30% solar, +50% demand, limited diesel, 6-hour grid outage on Day 2.* |
|
|
| | Rank | Agent | Score | |
| |---:|---|:---:| |
| | 1 | Grok-4 | **0.72** | |
| | 2 | Oracle | 0.70 | |
| | 3 | GPT-5.4 | 0.67 | |
| | 4 | Gemma-4-31B | 0.62 | |
| | 4 | DeepSeek V3.2 | 0.62 | |
| | 6 | Grok 4.20 Multi-Agent | 0.60 | |
| | 7 | Kimi K2.5 | 0.48 | |
| | 8 | Gemini 3.1 Pro Preview | 0.47 | |
| | 9 | GPT-5.4-mini | 0.46 | |
| | 10 | Qwen 3.6 Plus | 0.45 | |
| | 10 | Do-Nothing | 0.45 | |
| | 10 | Always-Discharge | 0.45 | |
| | 13 | Always-Diesel | 0.44 | |
|
|
| --- |
|
|
| ## Key Observations |
|
|
| 1. **The environment cleanly differentiates capability.** A clean gradient from `do-nothing` (0.51 avg) through frontier LLMs (0.78). Every model lands in a different tier. |
|
|
| 2. **Task 3 is the real differentiator.** The 6-hour grid outage forces true islanding behavior. Only Grok-4 and the Oracle handle it well (>0.70). Most LLMs collapse to ~0.45 — the same as do-nothing. |
|
|
| 3. **Frontier LLMs match or beat the hand-coded oracle.** Grok-4 (0.78) > Oracle (0.77) — the environment is solvable by raw LLM reasoning, but requires real intelligence. |
|
|
| 4. **Smaller LLMs barely beat do-nothing.** Kimi K2.5 (0.53) and Gemini 3.1 Pro Preview (0.55) are within rounding error of the do-nothing baseline (0.51) — they struggle to produce useful actions consistently. |
|
|
| 5. **Capability scales with model size within a family.** GPT-5.4 (0.75) significantly outperforms GPT-5.4-mini (0.64). Same prompt, same environment — only the model size differs. |
|
|
| 6. **The 0.20-0.35 gap between best and worst agents** proves the environment has real optimization headroom and isn't trivially solvable. |
|
|
| --- |
|
|
| ## Reproducibility |
|
|
| All scores are deterministic. To reproduce: |
|
|
| ```bash |
| export API_BASE_URL="https://openrouter.ai/api/v1" |
| export HF_TOKEN="<your-key>" |
| export MODEL_NAME="<model-id>" |
| python inference.py |
| ``` |
|
|
| Output is structured `[START] / [STEP] / [END]` blocks with explicit task names and scores. Same seed (42) + same model = identical scores across runs. |
|
|
| To run hand-coded baselines (no API key needed): |
|
|
| ```bash |
| python scripts/oracle_test.py |
| ``` |
|
|
