docs: rewrite and expand blog post describing GridMind-RL environment architecture and results

BLOG.md

@@ -1,6 +1,6 @@
 ---
 title: GridMind-RL: Training LLMs to Manage Industrial Buildings with GRPO
-description: How we built an OpenEnv-compatible RL environment that teaches language models real-world energy management.
+description: How we built an OpenEnv-compatible RL environment that teaches language models real-world energy management — and what the training curves actually show.
 ---
 
 # GridMind-RL: Training LLMs to Manage Industrial Buildings
@@ -9,60 +9,284 @@ description: How we built an OpenEnv-compatible RL environment that teaches lang
 
 ---
 
-Industrial buildings consume 40% of global electricity, yet most are managed by static, inefficient schedules. GridMind-RL bridges the gap between LLM reasoning and real-world action, training agents to manage energy costs, comfort, and grid stability under pressure. Our results show a trained agent beating hand-crafted heuristics by **58%** on complex, instruction-driven tasks.
+There is a building somewhere running its air conditioning at full power right now,
+even though electricity costs five times more than it did six hours ago. Not because
+the operator made a bad decision — but because the control system doesn't know the
+price changed.
 
-## Why it Matters
-We built GridMind-RL for the **Meta PyTorch OpenEnv Hackathon Grand Finale**. It directly addresses:
-- **Theme 1 (Multi-Agent):** Buildings share a grid feeder; actions in one affect stress for all.
-- **Theme 3 (World Modeling):** Agents use a `/simulate` endpoint to "think ahead" before acting.
+Industrial buildings consume roughly 40% of global electricity. Most are managed by
+fixed schedules that made sense when they were written and haven't been touched since.
+The cost gap between a naive policy and an intelligent one is measurable in thousands
+of dollars per building per year.
+
+LLMs can read pricing curves, respond to fault alerts, and follow natural language
+instructions — but there has never been an environment that trains them to *act* on
+that reasoning under real operational pressure. We built one, trained on it, and the
+results show an agent that beats a hand-crafted heuristic on the tasks that matter most.
+
+---
+
+## Who We Are
+
+We are a team of three fascinated by the gap between what LLMs can reason about and
+what they can actually *do*. Building energy management sits right at that frontier —
+the domain is rich, the stakes are real, and no RL benchmark has touched it.
+GridMind-RL is our attempt to change that.
+
+We built this for the Meta PyTorch OpenEnv Hackathon Grand Finale at Scaler School
+of Technology, Bangalore, April 25–26, 2026.
+
+---
+
+## Which Themes We're Targeting
+
+GridMind-RL directly addresses two hackathon themes:
+
+**Theme 1 — Multi-Agent Interactions:** Three buildings share a 360kW grid feeder
+(120kW per building). A coordinator LLM reads fleet-wide demand via `/feeder` and
+sets per-building price multipliers via `/coordinate`. Buildings that ignore the
+signal trip the feeder limit — causing a grid fault penalty for all three. This
+creates genuine emergent coordination pressure without explicit communication.
+
+**Theme 3.1 — World Modeling (Professional Tasks):** The `/simulate` endpoint lets
+the agent ask "what if?" before committing an action. When HVAC efficiency is low or
+faults are active, the agent can simulate a proposed action and revise its plan if
+the predicted reward is poor. This trains causal reasoning and persistent world
+modeling — exactly what Theme 3 targets.
+
+---
 
 ## The Environment
-GridMind-RL is an OpenEnv-compatible Go server simulating a 24-hour industrial building cycle (96 steps). Agents must navigate volatile prices, equipment faults, and grid stress signals.
 
-### Actions & Rewards
-| Action | Purpose |
-|--------|---------|
-| `hvac_power_level` | Climate control |
-| `thermal_charge_rate` | Energy storage arbitrage |
-| `batch_job_slot` | Industrial load scheduling |
-| `load_shed_fraction` | Demand response |
+GridMind-RL implements the OpenEnv-compatible interface (reset/step/state/grade)
+via a high-performance Go HTTP server. openenv-core==0.2.3 is used as the
+Python client library for training-side interaction. It simulates a complete 24-hour industrial
+building energy system at 15-minute resolution — 96 decision steps per episode.
+
+The agent operates in continuous time, responding to a world that changes around it:
+prices spike up to 5× during tariff faults, equipment degrades, grid stress signals
+arrive, and sometimes the chiller fails at 2pm on the hottest day of the year.
+
+**The agent sees a rich observation space every step, including:**
+indoor temperature, thermal storage level, electricity price, grid stress signal,
+HVAC efficiency (which degrades continuously throughout the episode), active fault
+alarms, a 4-step price forecast, cumulative cost, carbon intensity, batch job queue,
+and hour of day. In Task 4, this also includes a natural language instruction card.
+
+**The agent has four levers:**
+
+| Action | Range | What it does |
+|--------|-------|--------------|
+| `hvac_power_level` | 0 → 1 | How hard the HVAC system works |
+| `thermal_charge_rate` | -1 → 1 | Charge or discharge thermal storage |
+| `batch_job_slot` | 0 → 4 | When to run deferrable industrial loads |
+| `load_shed_fraction` | 0 → 0.5 | Voluntary demand reduction during grid stress |
+
+**Four tasks of increasing difficulty:**
+
+- **Cost Minimization** — Navigate 24-hour price volatility (~2¢ to ~36¢/kWh) and
+  thermal storage arbitrage to minimize total energy spend.
+
+- **Comfort Management** — Hold indoor temperature within 19–23°C through equipment
+  degradation, faults, and shifting external conditions.
+
+- **Demand Response** — Read grid stress signals in real time and voluntarily shed
+  load (when signal exceeds 0.7) to earn demand-response credit without sacrificing
+  comfort.
+
+- **Instruction Following** — Parse a natural language objective card at episode
+  start and adapt the entire 96-step strategy to meet it.
+
+### Why the reward has nine components
+
+The naive approach is to reward cost savings and call it done. The problem is that
+a cost-only reward teaches the agent to turn off the HVAC entirely — perfect score,
+frozen building. This is textbook reward hacking.
+
+Real building operators don't optimize one metric. They manage a hierarchy:
+comfort is non-negotiable, grid compliance is contractual, cost is the primary KPI,
+carbon is increasingly regulated, and equipment stability protects the capital budget.
+
+Our reward reflects that hierarchy directly:
+
+| Component | Weight | Why |
+|-----------|--------|-----|
+| `cost_savings` | 0.28 | Primary operator KPI |
+| `carbon_reward` | 0.20 | ESG compliance, increasingly mandatory |
+| `temp_constraint` | 0.20 | Hard safety constraint — SLA violations incur penalties |
+| `grid_response` | 0.20 | Demand response programs pay operators to shed load |
+| `batch_deadline` | 0.12 | Missing deadlines causes downstream production losses |
+| `efficiency_bonus` | 0.05 | Incentivises smart thermal storage arbitrage |
+| `stability_penalty` | -0.05 | Prevents HVAC thrashing that causes equipment wear |
+| `fault_mitigation` | dynamic | Correct fault response prevents costly outages |
+| `task_satisfaction` | 0.10–0.50* | Task 4 only — weighted per the instruction card |
+
+> *`task_satisfaction` weight varies by instruction template, ranging from
+> 0.10 to 0.50 depending on the episode's objective card (tasks.go).
+
+### How we prevent reward hacking
+
+A multi-component reward is only part of the answer. We also:
+
+- **Clamp all actions** at the server side — the agent cannot exceed valid ranges
+  regardless of what it outputs (`hvac_power_level` hard-clamped 0–1,
+  `load_shed_fraction` hard-clamped 0–0.5, etc.)
+- **Inject four fault types** that make naive exploitation brittle: chiller failure
+  (HVAC drops to 20% capacity), grid outage (price up to ×4, stress = 1.0), sensor
+  fault (temperature jitter ±5°C), and tariff spike (price up to ×5)
+- **Use a seeded but stochastic environment** — price curves, fault timing, and
+  demand patterns vary across episodes, preventing the agent from memorizing a
+  fixed solution
+- **Score via `/grade`** at episode end using a separate grading function that is
+  decoupled from the per-step reward signal
+
+---
+
+## Training
+
+We trained Qwen2.5-1.5B-Instruct with QLoRA (4-bit, rank 16) using GRPO via
+HuggingFace TRL on a T4 GPU — roughly 35 minutes per run.
+
+| Component | Detail |
+|-----------|--------|
+| Model | Qwen2.5-1.5B-Instruct |
+| Fine-tuning | QLoRA (4-bit, rank 16) |
+| Algorithm | GRPO via HuggingFace TRL |
+| Hardware | HF Space T4 GPU |
+| Training time | ~35 minutes |
+| Steps | 60 |
+
+**Why GRPO over PPO?**
+GRPO doesn't require a separate value network. At 1.5B parameters on a T4, that
+memory saving matters. Instead of estimating a value baseline, GRPO samples a group
+of completions per prompt and computes advantages by comparing them against each
+other — a natural fit for our setting where we generate multiple actions per state
+and want to reinforce the better ones.
+
+The hackathon context emphasized that RL only works if the probability of a good
+answer is greater than zero. We confirmed this by running a heuristic baseline first
+to verify the environment produces non-zero reward before starting RL training.
+
+---
+
+## Results
+
+### The numbers first
 
-**The Reward System:** A 9-component verifiable reward (no LLM judge) balancing cost (28%), carbon (20%), comfort (20%), grid response (20%), and task satisfaction.
+| Policy | Task 1 | Task 2 | Task 3 | Task 4 | Avg (unweighted) |
+|--------|--------|--------|--------|--------|------------------|
+| Heuristic Baseline | 0.54 | 0.56 | 0.50 | 0.31 | 0.48 |
+| GRPO Fine-tuned | 0.42 | 0.34 | 0.47 | **0.49** | 0.43 |
 
-## Training & Results
-We trained **Qwen2.5-1.5B-Instruct** using **GRPO via TRL** with QLoRA (4-bit) on a T4 GPU.
+> Heuristic = fixed time-of-day HVAC scheduling, no learning.
+> GRPO Fine-tuned = Qwen2.5-1.5B-Instruct after 60 steps of GRPO training
+> against the live environment.
 
-### The Results
-| Policy | Task 1 (Cost) | Task 4 (Instruction) |
-|--------|--------|------------------|
-| Heuristic Baseline | 0.54 | 0.31 |
-| **GRPO Fine-tuned** | 0.42 | **0.49 (+58%)** |
+The trained model **beats the heuristic on Task 4 by 58%** (0.49 vs 0.31) and
+**comes within 6% of the heuristic on Task 3** (0.47 vs 0.50).
 
-While heuristics are strong on simple scheduling, the **GRPO-trained model dominates Task 4**, proving LLMs can parse complex natural language objectives and adapt strategies mid-episode—a feat impossible for fixed schedules.
+These are the two tasks where intelligent reasoning matters most — instruction
+parsing and real-time grid cooperation. A fixed schedule cannot read an objective
+card. A fixed schedule cannot respond to a grid stress signal that arrives mid-episode.
+The trained model can do both.
+
+Tasks 1 and 2 are an honest result. Time-of-day HVAC scheduling is genuinely
+competitive for cost and comfort — the heuristic baseline is strong on those
+objectives because the physics are predictable. Closing that gap requires more
+training steps. The reward curve shows the trend is still moving upward at step 60,
+meaning training had not plateaued.
+
+### The reward curve
 
 ![Reward Curve](curves/train%204/reward_curve.png)
-*Learning progress: Reward climbed from −0.47 to +0.61 in just 60 steps, with the curve still rising at termination.*
+*Reward vs training step. From −0.47 at step 5 to +0.61 at step 60 — a 1.08-point
+gain. The smoothed average (red dashed) is still rising at the final step, confirming
+training had not saturated.*
+
+### The before/after
+
+![Baseline Comparison](curves/train%204/baseline_comparison.png)
+*Grade scores per task: heuristic baseline (blue) vs GRPO-trained LLM (green).
+Task 4 is where the trained model pulls clearly ahead — 58% above the heuristic.*
+
+---
+
+## What the Agent Learns
+
+None of these behaviors are hardcoded. The reward signal surfaces them:
 
-## Future Directions
-- **Scaling:** Moving to 7B+ models for deeper planning.
-- **Fleet Coordination:** Orchestrating building clusters to avoid feeder trips.
-- **Real-world Bridge:** Deploying the physics-grounded simulator to live building management systems.
+**Thermal arbitrage** — the agent learns to charge thermal storage during off-peak
+hours (~3.5¢/kWh) and discharge during peak (~31¢/kWh), reducing the effective cost
+of maintaining comfort during expensive periods.
+
+**Grid cooperation** — when the stress signal exceeds 0.7, the agent voluntarily
+sheds load rather than ignoring it. The demand-response credit offsets the comfort
+penalty — which is why Task 3 performance is closest to the heuristic.
+
+**Fault adaptation** — when HVAC efficiency degrades, the agent reduces its HVAC
+target rather than fighting a weakened system at full power. This behavior emerges
+purely from the `fault_mitigation` reward component.
+
+**Instruction parsing** — in Task 4, the agent reads the objective card and adjusts
+its entire 96-step strategy to meet it. This is the hardest capability for a
+heuristic to replicate — and where the trained model wins most clearly.
+
+---
+
+## What's Next
+
+GridMind-RL is a foundation, not a finished product. The directions we find most
+interesting:
+
+**Longer training runs** — the reward curve hasn't plateaued at 60 steps. 300+
+steps would likely close the gap on Tasks 1 and 2 and push Task 4 performance
+further above the heuristic.
+
+**Larger models** — a 7B model with the same training setup would bring stronger
+instruction-following capability and better multi-step planning out of the box.
+
+**Fleet-level coordination** — three buildings share a 360kW grid feeder (120kW per
+building). Fleet-level coordination is fully implemented — training a coordinator LLM
+that orchestrates all three through price signals is the next research direction.
+The shared feeder constraint creates genuine emergent coordination pressure — if one
+building ignores the signal, all three pay the penalty.
+
+**Real deployment** — the environment's physics are grounded in real building
+parameters. The gap between this simulator and a real BMS integration is smaller
+than it looks.
+
+---
 
 ## Try It
+
+GridMind-RL is live and OpenEnv-compliant. Task 4 is the most interesting to try —
+the agent receives a natural language objective card and must adapt its entire
+strategy to meet it:
+
 ```bash
+# Health check
+curl https://prajwal782007-gridmind.hf.space/health
+
 # Start a Task 4 episode (instruction following)
-curl -X POST https://prajwal782007-gridmind.hf.space/reset -d '{"task_id": 4}'
+curl -X POST https://prajwal782007-gridmind.hf.space/reset \
+  -H "Content-Type: application/json" \
+  -d '{"task_id": 4}'
 
 # Take an action and observe the reward
-curl -X POST https://prajwal782007-gridmind.hf.space/step -d '{"hvac_power_level": 0.6, "building_id": 0}'
+curl -X POST https://prajwal782007-gridmind.hf.space/step \
+  -H "Content-Type: application/json" \
+  -d '{"hvac_power_level": 0.6, "thermal_charge_rate": 0.4,
+       "batch_job_slot": 2, "load_shed_fraction": 0.0, "building_id": 0}'
 
 # Grade the full episode
 curl https://prajwal782007-gridmind.hf.space/grade
 ```
 
-- 🤗 **Environment**: [HF Space](https://prajwal782007-gridmind.hf.space)
-- 📓 **Training**: [Colab Notebook](https://colab.research.google.com/github/LO-Kyu/gridmind/blob/main/scripts/gridmind_grpo_colab.ipynb)
-- 🐙 **Code**: [GitHub Repository](https://github.com/LO-Kyu/gridmind)
+- 🤗 **Environment**: https://prajwal782007-gridmind.hf.space
+- 📓 **Training Notebook**: [gridmind_grpo_colab.ipynb](https://colab.research.google.com/github/LO-Kyu/gridmind/blob/main/scripts/gridmind_grpo_colab.ipynb)
+- 🐙 **Code**: https://github.com/LO-Kyu/gridmind
 
 ---
-*Built for the Meta PyTorch OpenEnv Hackathon · Grand Finale · Bangalore 2026*
+
+*Built for the Meta PyTorch OpenEnv Hackathon × Scaler School of Technology ·
+Grand Finale, April 25–26, 2026, Bangalore.*