docs: update HF_BLOG_POST.md with OpenEnv interface details, reward engineering insights, and refined training results

HF_BLOG_POST.md

@@ -1,16 +1,16 @@
 ---
 title: GridMind-RL: Training LLMs to Manage Industrial Buildings with GRPO
-description: How we built an RL environment that teaches language models real-world energy management — and what 10 training runs taught us.
+description: How we built an OpenEnv-compatible RL environment that teaches language models real-world energy management — and what the training curves actually show.
 ---
 
 # GridMind-RL: Training LLMs to Manage Industrial Buildings
 
-*OpenEnv Hackathon India 2026 · GridMind-RL Team*
+*OpenEnv Hackathon India 2026 · Aditya Suryavanshi, Shreeshant Bokade, Prajwal Valekar*
 
 ---
 
 There is a building somewhere running its air conditioning at full power right now,
-even though electricity costs four times more than it did six hours ago. Not because
+even though electricity costs five times more than it did six hours ago. Not because
 the operator made a bad decision — but because the control system doesn't know the
 price changed.
 
@@ -20,27 +20,58 @@ The cost gap between a naive policy and an intelligent one is measurable in thou
 of dollars per building per year.
 
 LLMs can read pricing curves, respond to fault alerts, and follow natural language
-instructions. The missing piece has always been an environment that trains them to
-*act* on that reasoning under real operational pressure.
+instructions — but there has never been an environment that trains them to *act* on
+that reasoning under real operational pressure. We built one, trained on it, and the
+results show an agent that beats a hand-crafted heuristic on the tasks that matter most.
 
-That's what we built.
+---
+
+## Who We Are
+
+We are a team of three fascinated by the gap between what LLMs can reason about and
+what they can actually *do*. Building energy management sits right at that frontier —
+the domain is rich, the stakes are real, and no RL benchmark has touched it.
+GridMind-RL is our attempt to change that.
+
+We built this for the Meta PyTorch OpenEnv Hackathon Grand Finale at Scaler School
+of Technology, Bangalore, April 25–26, 2026.
+
+---
+
+## Which Themes We're Targeting
+
+GridMind-RL directly addresses two hackathon themes:
+
+**Theme 1 — Multi-Agent Interactions:** Three buildings share a 360kW grid feeder
+(120kW per building). A coordinator LLM reads fleet-wide demand via `/feeder` and
+sets per-building price multipliers via `/coordinate`. Buildings that ignore the
+signal trip the feeder limit — causing a grid fault penalty for all three. This
+creates genuine emergent coordination pressure without explicit communication.
+
+**Theme 3.1 — World Modeling (Professional Tasks):** The `/simulate` endpoint lets
+the agent ask "what if?" before committing an action. When HVAC efficiency is low or
+faults are active, the agent can simulate a proposed action and revise its plan if
+the predicted reward is poor. This trains causal reasoning and persistent world
+modeling — exactly what Theme 3 targets.
 
 ---
 
 ## The Environment
 
-GridMind-RL simulates a complete 24-hour industrial building energy system at
-15-minute resolution — 96 decision steps per episode. The agent operates in
-continuous time, responding to a world that changes around it: prices spike, equipment
-degrades, grid stress signals arrive, and sometimes the chiller fails at 2pm on the
-hottest day of the year.
+GridMind-RL implements the OpenEnv-compatible interface (reset/step/state/grade)
+via a high-performance Go HTTP server. openenv-core==0.2.3 is used as the
+Python client library for training-side interaction. It simulates a complete 24-hour industrial
+building energy system at 15-minute resolution — 96 decision steps per episode.
+
+The agent operates in continuous time, responding to a world that changes around it:
+prices spike up to 5× during tariff faults, equipment degrades, grid stress signals
+arrive, and sometimes the chiller fails at 2pm on the hottest day of the year.
 
-**The agent sees 13 fields every step:**
-current indoor temperature, thermal storage level, electricity price, grid stress
-signal, HVAC efficiency (which degrades continuously over the episode), active fault
-alarms, a 4-step price forecast, cumulative cost so far, carbon intensity, batch job
-queue, hour of day, and — in Task 4 — a natural language instruction card describing
-the episode's objective.
+**The agent sees a rich observation space every step, including:**
+indoor temperature, thermal storage level, electricity price, grid stress signal,
+HVAC efficiency (which degrades continuously throughout the episode), active fault
+alarms, a 4-step price forecast, cumulative cost, carbon intensity, batch job queue,
+and hour of day. In Task 4, this also includes a natural language instruction card.
 
 **The agent has four levers:**
 
@@ -51,16 +82,17 @@ the episode's objective.
 | `batch_job_slot` | 0 → 4 | When to run deferrable industrial loads |
 | `load_shed_fraction` | 0 → 0.5 | Voluntary demand reduction during grid stress |
 
-**Four tasks test different capabilities:**
+**Four tasks of increasing difficulty:**
 
-- **Cost Minimization** — Navigate 24-hour price volatility and thermal storage
-  arbitrage to minimize total energy spend.
+- **Cost Minimization** — Navigate 24-hour price volatility (~2¢ to ~36¢/kWh) and
+  thermal storage arbitrage to minimize total energy spend.
 
 - **Comfort Management** — Hold indoor temperature within 19–23°C through equipment
   degradation, faults, and shifting external conditions.
 
 - **Demand Response** — Read grid stress signals in real time and voluntarily shed
-  load to earn demand-response credit without sacrificing comfort.
+  load (when signal exceeds 0.7) to earn demand-response credit without sacrificing
+  comfort.
 
 - **Instruction Following** — Parse a natural language objective card at episode
   start and adapt the entire 96-step strategy to meet it.
@@ -69,7 +101,7 @@ the episode's objective.
 
 The naive approach is to reward cost savings and call it done. The problem is that
 a cost-only reward teaches the agent to turn off the HVAC entirely — perfect score,
-frozen building.
+frozen building. This is textbook reward hacking.
 
 Real building operators don't optimize one metric. They manage a hierarchy:
 comfort is non-negotiable, grid compliance is contractual, cost is the primary KPI,
@@ -87,24 +119,33 @@ Our reward reflects that hierarchy directly:
 | `efficiency_bonus` | 0.05 | Incentivises smart thermal storage arbitrage |
 | `stability_penalty` | -0.05 | Prevents HVAC thrashing that causes equipment wear |
 | `fault_mitigation` | dynamic | Correct fault response prevents costly outages |
-| `task_satisfaction` | 0.50* | Task 4 only — weighted per the instruction card |
+| `task_satisfaction` | 0.10–0.50* | Task 4 only — weighted per the instruction card |
+
+> *`task_satisfaction` weight varies by instruction template, ranging from
+> 0.10 to 0.50 depending on the episode's objective card (tasks.go).
+
+### How we prevent reward hacking
+
+A multi-component reward is only part of the answer. We also:
 
-A reward this dense is harder to game. An agent that exploits one component while
-neglecting the others will see it reflected immediately in the score.
+- **Clamp all actions** at the server side — the agent cannot exceed valid ranges
+  regardless of what it outputs (`hvac_power_level` hard-clamped 0–1,
+  `load_shed_fraction` hard-clamped 0–0.5, etc.)
+- **Inject four fault types** that make naive exploitation brittle: chiller failure
+  (HVAC drops to 20% capacity), grid outage (price up to ×4, stress = 1.0), sensor
+  fault (temperature jitter ±5°C), and tariff spike (price up to ×5)
+- **Use a seeded but stochastic environment** — price curves, fault timing, and
+  demand patterns vary across episodes, preventing the agent from memorizing a
+  fixed solution
+- **Score via `/grade`** at episode end using a separate grading function that is
+  decoupled from the per-step reward signal
 
 ---
 
 ## Training
 
 We trained Qwen2.5-1.5B-Instruct with QLoRA (4-bit, rank 16) using GRPO via
-HuggingFace TRL. Each run is 60 steps on a T4 GPU, taking roughly 35 minutes.
-We ran 10 training iterations in total.
-
-**Why GRPO over PPO?**
-GRPO doesn't require a separate value network. At 1.5B parameters on a T4, that
-memory saving matters. Instead of estimating a value baseline, GRPO samples a group
-of completions per prompt and computes advantages by comparing them against each
-other — a natural fit for our setting where we generate multiple actions per state.
+HuggingFace TRL on a T4 GPU — roughly 35 minutes per run.
 
 | Component | Detail |
 |-----------|--------|
@@ -112,67 +153,61 @@ other — a natural fit for our setting where we generate multiple actions per s
 | Fine-tuning | QLoRA (4-bit, rank 16) |
 | Algorithm | GRPO via HuggingFace TRL |
 | Hardware | HF Space T4 GPU |
-| Training time | ~35 minutes per run |
-| Total runs | 10 |
-
----
-
-## What the Curves Show
-
-### Run 1 vs Run 10: The reward is climbing
+| Training time | ~35 minutes |
+| Steps | 60 |
 
-The clearest evidence of learning is what happens to the reward curve within a single
-training run — and how that shape changes as the training setup matures.
-
-**Run 1 — the first training run:**
-
-![Reward Curve — Run 1](curves/train%201/reward_curve.png)
-*Run 1: Reward climbs from −0.47 to ~0.65 over 60 steps. The model is learning fast
-in the early steps, then stabilizing — with a small dip at the very end.*
+**Why GRPO over PPO?**
+GRPO doesn't require a separate value network. At 1.5B parameters on a T4, that
+memory saving matters. Instead of estimating a value baseline, GRPO samples a group
+of completions per prompt and computes advantages by comparing them against each
+other — a natural fit for our setting where we generate multiple actions per state
+and want to reinforce the better ones.
 
-**Run 10 — after iterative refinement:**
+The hackathon context emphasized that RL only works if the probability of a good
+answer is greater than zero. We confirmed this by running a heuristic baseline first
+to verify the environment produces non-zero reward before starting RL training.
 
-![Reward Curve — Run 10](curves/train%2010/reward_curve.png)
-*Run 10: Same starting point, smoother curve, still rising at step 60. The model
-hasn't plateaued — which means longer training would continue to improve it.*
+---
 
-Both runs start at the same reward (~−0.47) because each run initializes fresh.
-What changes is the *shape*: Run 10 is more stable, ends higher (~0.68 vs ~0.65),
-and shows no end-of-run dip. Ten runs of iteration on the training setup produced
-a meaningfully cleaner learning signal.
+## Results
 
-The 1.1-point reward improvement within a single 60-step run is not noise.
-The agent is learning to manage energy in real time.
+### The numbers first
 
-### Before and After: Where the model wins
+| Policy | Task 1 | Task 2 | Task 3 | Task 4 | Avg (unweighted) |
+|--------|--------|--------|--------|--------|------------------|
+| Heuristic Baseline | 0.54 | 0.56 | 0.50 | 0.31 | 0.48 |
+| GRPO Fine-tuned | 0.42 | 0.34 | 0.47 | **0.49** | 0.43 |
 
-**Run 1 — heuristic baseline vs GRPO-trained:**
+> Heuristic = fixed time-of-day HVAC scheduling, no learning.
+> GRPO Fine-tuned = Qwen2.5-1.5B-Instruct after 60 steps of GRPO training
+> against the live environment.
 
-![Baseline Comparison — Run 1](curves/train%201/baseline_comparison.png)
-*Run 1: The trained model outperforms the heuristic on Task 4 by a significant margin.
-On Tasks 1–3 it scores below the heuristic — early training, limited steps.*
+The trained model **beats the heuristic on Task 4 by 58%** (0.49 vs 0.31) and
+**comes within 6% of the heuristic on Task 3** (0.47 vs 0.50).
 
-**Run 10 — heuristic baseline vs GRPO-trained:**
+These are the two tasks where intelligent reasoning matters most — instruction
+parsing and real-time grid cooperation. A fixed schedule cannot read an objective
+card. A fixed schedule cannot respond to a grid stress signal that arrives mid-episode.
+The trained model can do both.
 
-![Baseline Comparison — Run 10](curves/train%2010/baseline_comparison.png)
-*Run 10: Similar pattern. Task 4 remains the trained model's strongest result.
-Tasks 1–3 gap to the heuristic has narrowed compared to Run 1.*
+Tasks 1 and 2 are an honest result. Time-of-day HVAC scheduling is genuinely
+competitive for cost and comfort — the heuristic baseline is strong on those
+objectives because the physics are predictable. Closing that gap requires more
+training steps. The reward curve shows the trend is still moving upward at step 60,
+meaning training had not plateaued.
 
-### The Task 4 result is the headline
+### The reward curve
 
-The heuristic scores **0.30** on Task 4. The trained model scores **0.70**.
-That is a **133% improvement** on instruction following — and it makes complete sense.
+![Reward Curve](curves/train%204/reward_curve.png)
+*Reward vs training step. From −0.47 at step 5 to +0.61 at step 60 — a 1.08-point
+gain. The smoothed average (red dashed) is still rising at the final step, confirming
+training had not saturated.*
 
-A fixed heuristic cannot read a natural language objective card. It cannot parse
-"keep total cost under $2.50 while maintaining comfort" and change its behavior
-accordingly. The trained model can. That capability gap is exactly what this
-environment was designed to measure.
+### The before/after
 
-Tasks 1–3 tell a more honest story. Time-of-day HVAC scheduling is genuinely
-reasonable for cost and temperature — the heuristic is a strong baseline on those
-tasks, and 60 training steps with a 1.5B model isn't enough to beat it consistently.
-That's not a failure of the environment. It's a signal that longer training would
-continue to pay off.
+![Baseline Comparison](curves/train%204/baseline_comparison.png)
+*Grade scores per task: heuristic baseline (blue) vs GRPO-trained LLM (green).
+Task 4 is where the trained model pulls clearly ahead — 58% above the heuristic.*
 
 ---
 
@@ -181,50 +216,70 @@ continue to pay off.
 None of these behaviors are hardcoded. The reward signal surfaces them:
 
 **Thermal arbitrage** — the agent learns to charge thermal storage during off-peak
-hours (~4¢/kWh) and discharge during peak (~32¢/kWh), reducing the effective cost
+hours (~3.5¢/kWh) and discharge during peak (~31¢/kWh), reducing the effective cost
 of maintaining comfort during expensive periods.
 
 **Grid cooperation** — when the stress signal exceeds 0.7, the agent voluntarily
-sheds load rather than ignoring the signal. The demand-response credit offsets the
-comfort penalty.
+sheds load rather than ignoring it. The demand-response credit offsets the comfort
+penalty — which is why Task 3 performance is closest to the heuristic.
 
-**Fault adaptation** — when HVAC efficiency degrades below a threshold, the agent
-reduces its HVAC target rather than fighting a weakened system at full power.
+**Fault adaptation** — when HVAC efficiency degrades, the agent reduces its HVAC
+target rather than fighting a weakened system at full power. This behavior emerges
+purely from the `fault_mitigation` reward component.
 
 **Instruction parsing** — in Task 4, the agent reads the objective card and adjusts
-its entire 96-step strategy accordingly, not just the next action.
+its entire 96-step strategy to meet it. This is the hardest capability for a
+heuristic to replicate — and where the trained model wins most clearly.
 
 ---
 
-## What We'd Do With More Compute
+## What's Next
+
+GridMind-RL is a foundation, not a finished product. The directions we find most
+interesting:
 
-- **300+ training steps** would likely close the gap on Tasks 1–3
-- A **7B model** with the same setup would show sharper policy improvement
-- **Multi-agent coordination** — 3 buildings sharing a 250kW feeder — is fully
-  implemented but not yet the primary training focus. Fleet-level demand response
-  is the next frontier.
+**Longer training runs** — the reward curve hasn't plateaued at 60 steps. 300+
+steps would likely close the gap on Tasks 1 and 2 and push Task 4 performance
+further above the heuristic.
+
+**Larger models** — a 7B model with the same training setup would bring stronger
+instruction-following capability and better multi-step planning out of the box.
+
+**Fleet-level coordination** — three buildings share a 360kW grid feeder (120kW per
+building). Fleet-level coordination is fully implemented — training a coordinator LLM
+that orchestrates all three through price signals is the next research direction.
+The shared feeder constraint creates genuine emergent coordination pressure — if one
+building ignores the signal, all three pay the penalty.
+
+**Real deployment** — the environment's physics are grounded in real building
+parameters. The gap between this simulator and a real BMS integration is smaller
+than it looks.
 
 ---
 
 ## Try It
 
-The environment is live. You can reset an episode, send actions, and read rewards
-right now from your terminal:
+GridMind-RL is live and OpenEnv-compliant. Task 4 is the most interesting to try —
+the agent receives a natural language objective card and must adapt its entire
+strategy to meet it:
 
 ```bash
 # Health check
 curl https://prajwal782007-gridmind.hf.space/health
 
-# Start an episode
+# Start a Task 4 episode (instruction following)
 curl -X POST https://prajwal782007-gridmind.hf.space/reset \
   -H "Content-Type: application/json" \
   -d '{"task_id": 4}'
 
-# Take an action
+# Take an action and observe the reward
 curl -X POST https://prajwal782007-gridmind.hf.space/step \
   -H "Content-Type: application/json" \
   -d '{"hvac_power_level": 0.6, "thermal_charge_rate": 0.4,
        "batch_job_slot": 2, "load_shed_fraction": 0.0, "building_id": 0}'
+
+# Grade the full episode
+curl https://prajwal782007-gridmind.hf.space/grade
 ```
 
 - 🤗 **Environment**: https://prajwal782007-gridmind.hf.space
@@ -233,5 +288,5 @@ curl -X POST https://prajwal782007-gridmind.hf.space/step \
 
 ---
 
-*Built for the OpenEnv Hackathon India 2026 · April 25–26 · Scaler School of
-Technology, Bangalore.*
+*Built for the Meta PyTorch OpenEnv Hackathon × Scaler School of Technology ·
+Grand Finale, April 25–26, 2026, Bangalore.*