Updated Readme for ROund 2

README.md

@@ -21,7 +21,7 @@ license: mit
 
 ## Why This Environment Is Novel
 
-Most RL environments for LLMs are grid-worlds or toy games. GridMind-RL simulates a **real industrial problem** — building energy management — where agents must juggle stochastic electricity prices, multi-objective constraints, equipment faults, and natural language operating objectives. An LLM that learns to manage a building under these conditions has a genuinely useful skill, not just a high game score.
+Industrial buildings consume ~40% of global electricity yet rely on naive "always-on" HVAC policies. LLMs can reason about pricing curves, fault alerts, and natural language objectives—but no environment trains them for this. GridMind-RL simulates a full 24-hour building energy system with stochastic electricity prices, equipment faults, and instruction cards, creating a genuinely challenging domain where learned policies translate to real operational value.
 
 ## Live Demo
 
@@ -38,24 +38,11 @@ curl https://prajwal782007-gridmind.hf.space/tasks
 
 ---
 
-## Problem
-
-Industrial buildings consume ~40% of global electricity, yet most still use naive "always-on" HVAC policies. The capability gap is clear: **LLMs can understand complex pricing curves, natural language instructions, and fault alerts—but no environment exists to train them to manage buildings.**
-
-GridMind-RL closes this gap by simulating a complete building energy system where agents must:
-- Navigate 24-hour price volatility (off-peak vs peak: 4¢ to 32¢/kWh)
-- Maintain comfort (19-23°C) while minimizing cost
-- Respond to grid stress emergencies
-- Handle equipment faults (chiller failure, sensor malfunction, grid outages)
-- Parse and follow natural language objective cards
-
----
-
 ## Environment
 
 | | Description |
 |---|-------------|
-| **Observation** | 11 fields: temperature, storage, price, stress, carbon, faults, HVAC efficiency |
+| **Observation** | 13 fields: temperature, storage, price, stress, carbon, faults, HVAC efficiency, process demand, batch queue, price forecast |
 | **Actions** | HVAC level (0-1), thermal charge (-1 to 1), batch slot (0-4), load shed (0-0.5) |
 | **Reward** | 9-component weighted sum: cost, temperature, grid, deadline, efficiency, stability, carbon, instruction, fault_mitigation |
 | **Episode** | 96 steps = 24 simulated hours @ 15-min resolution |
@@ -74,8 +61,8 @@ Weights reflect real-world building operator priorities — not arbitrary values
 | `batch_deadline` | 0.12 | Production continuity — missing batch deadlines causes downstream losses |
 | `efficiency_bonus` | 0.05 | Storage arbitrage — incentivises smart charge/discharge timing |
 | `stability_penalty` | -0.05 | Anti-cycling — prevents HVAC thrashing that causes equipment wear |
-| `fault_mitigation` | 0.05 | Emergency response — correct fault handling prevents costly outages |
-| `instruction_reward` | 0.50* | Task 4 only — weighted per the episode's instruction card |
+| `task_satisfaction` | 0.50* | Task 4 only — weighted per the episode's instruction card |
+| `fault_mitigation` | dynamic | Emergency response — computed based on fault type and response |
 
 > *Task 4 instruction reward weight comes from the sampled instruction card, not a fixed value.
 
@@ -85,11 +72,17 @@ Weights reflect real-world building operator priorities — not arbitrary values
 |-------|------|-------------|
 | indoor_temperature | float | °C |
 | thermal_storage_level | float | 0-1 (0=empty, 1=full) |
+| process_demand | float | kW current industrial power demand |
 | current_price | float | $/kWh |
 | grid_stress_signal | float | 0-1 (>0.7 = critical) |
+| carbon_intensity | float | gCO2/kWh |
+| hour_of_day | int | 0-23 |
+| batch_queue | int[] | pending job deadline slots |
+| cumulative_cost | float | $ total incurred this episode |
 | hvac_efficiency | float | 1.0 → degrades to 0.5 over episode |
 | active_faults | string[] | Active fault alarm strings |
 | instruction_card | object | Task 4 objective only |
+| price_forecast | float[] | 4-step upcoming price preview |
 
 ### Action Fields
 
@@ -102,41 +95,46 @@ Weights reflect real-world building operator priorities — not arbitrary values
 
 ---
 
-## Five Tracks
+## Core Capabilities
 
-### Track 1: Multi-Agent Interactions
+### Multi-Agent Coordination
 A single oversight LLM coordinates multiple buildings through price signals. The coordinator reads `/feeder` to see fleet-wide demand, then sets per-building price multipliers via `/coordinate` to orchestrate behavior.
 
-### Track 2: Long-Horizon Planning & Instruction Following
+### Long-Horizon Instruction Following
 Task 4 presents a natural language objective card like "Keep total energy cost under $2.50 while maintaining 19-23°C". Agents must plan across all 96 steps—not greedy per-step control.
 
-### Track 3: World Modeling
-The `/simulate` endpoint lets agents ask "what if?" before acting. When HVAC efficiency is low or faults are active, the agent simulates the proposed action and revises if the predicted reward is poor.
-
-### Track 4: Fault Handling (Wild Card)
-Four fault types inject unpredictability:
-- **Chiller failure**: HVAC drops to 20% capacity
-- **Grid outage**: Price ×3, stress = 1.0
-- **Sensor fault**: Temperature readings jitter ±5°C
-- **Tariff spike**: Emergency 4× price surge
-
-### Track 5: HVAC Degradation
-Real HVAC systems degrade over time. Efficiency starts at 1.0 and drops ~0.1% per step. The agent must account for declining capacity—a hidden state requiring inference.
+These two capabilities map directly to Theme 1 and Theme 3 of the OpenEnv Hackathon.
 
 ---
 
 ## Results
 
-![Training Curve](results/training_curve.png)
-*Episode grade scores vs training step. Heuristic baseline (red) vs GRPO fine-tuned LLM (teal). Higher = better energy management.*
+### What the Agent Learns
+
+A naive heuristic runs HVAC at fixed levels based on time-of-day. After GRPO training on GridMind-RL, the agent learns to charge thermal storage during off-peak hours (4¢/kWh) and discharge during peak (32¢/kWh), voluntarily shed load during grid stress signals above 0.7, and adjust HVAC intensity as efficiency degrades over the episode. None of these behaviors are hardcoded — the agent discovers them through the reward signal alone.
 
 | Policy | Task 1 | Task 2 | Task 3 | Task 4 |
 |--------|--------|--------|--------|--------|
-| Heuristic Baseline | 0.506 | 0.459 | 0.600 | 0.492 |
+| Heuristic Baseline | 0.494 | 0.471 | 0.748 | 0.478 |
 | Zero-shot LLM | 0.715 | 0.645 | 0.610 | 0.582 |
-| GRPO Fine-tuned LLM | TBD | TBD | TBD | TBD |
+| GRPO Fine-tuned LLM | — | — | — | — |
+
+> *GRPO fine-tuned scores updating after full training run on T4 GPU.
+> Training plots below show live progress from the actual run.*
 
-> Scores are episode grade scores (0.0–1.0, clamped open interval). Heuristic = fixed policy with no learning. Zero-shot = pretrained Qwen2.5-7B-Instruct. Fine-tuned = GRPO-trained on GridMind-RL environment.
+![Reward Curve](curves/train%202/reward_curve.png)
+*Reward vs training step. Blue = per-step reward, red dashed = smoothed average.*
+
+![Loss Curve](curves/train%202/loss_curve.png)
+*Training loss decreasing over steps — confirms the model is updating.*
+
+![Baseline Comparison](curves/train%202/baseline_comparison.png)
+*Grade scores per task: heuristic baseline vs GRPO-trained LLM.*
+
+> Scores are episode grade scores (0.0–1.0, clamped open interval). Heuristic = fixed policy with no learning. Zero-shot = Qwen2.5-1.5B-Instruct prompted with task description, no fine-tuning, evaluated over 1 episode per task. Fine-tuned = GRPO-trained on GridMind-RL environment.
+
+> 🔄 **Live update:** GRPO fine-tuned scores will be filled in here immediately
+> after the final training run completes on the T4 GPU.
 
 ---
 
@@ -186,18 +184,6 @@ python scripts/plot_results.py
 
 ---
 
-## Self-Improvement: Curriculum Learning
-
-The `--curriculum` flag enables automatic task progression:
-- Agent starts on Task 1 (easy)
-- After 5 episodes with average reward ≥ 0.55, advances to Task 2
-- After 5 episodes with average reward ≥ 0.50, advances to Task 3
-- After 5 episodes with average reward ≥ 0.45, advances to Task 4
-
-This directly targets the Self-Improvement hackathon theme.
-
----
-
 ## Architecture
 
 ```
@@ -230,11 +216,15 @@ Web Dashboard (dashboard/server.py) → Port 7861
 | GET | /state | Get current state |
 | GET | /grade | Grade episode (0.0-1.0 score) |
 | GET | /tasks | Available tasks |
-| GET | /metrics | System metrics |
+| GET | /metrics | Prometheus metrics |
 | GET | /replay | Episode history |
 | GET | /feeder | Aggregate fleet state |
 | POST | /coordinate | Set price multipliers |
 | POST | /simulate | World model prediction |
+| POST | /coordinator/reset | Reset multi-building episode |
+| POST | /coordinator/step | Step with per-building actions |
+| GET | /info | OpenEnv metadata |
+| GET | /ws | WebSocket endpoint |
 
 ---
 
@@ -246,6 +236,8 @@ gridmind-rl/
 ├── inference.py              # Agent entry point (LLM + heuristic)
 ├── openenv.yaml              # OpenEnv spec
 ├── Dockerfile                # Container build
+├── HF_BLOG_POST.md           # Blog write-up
+├── baseline_scores.json      # Heuristic baseline scores
 ├── env/
 │   ├── environment.go        # Physics simulation
 │   ├── models.go           # Data models
@@ -256,10 +248,14 @@ gridmind-rl/
 │   ├── train_unsloth.py   # GRPO training
 │   ├── plot_results.py   # Training curve visualizer
 │   ├── multi_building_demo.py  # Fleet AI demo
-│   └── run_baseline.sh   # Baseline scorer
+│   └── gridmind_grpo_colab.ipynb  # Colab training notebook
+├── server/
+│   └── app.py            # Python fallback server
 ├── dashboard/
 │   ├── server.py         # Web server (port 7861)
 │   └── static/           # Frontend assets
+├── curves/               # Training curves (train N/)
+│   └── train N/         # Per-run plots
 ├── results/              # Training outputs (generated)
 └── README.md
 ```
@@ -269,9 +265,9 @@ gridmind-rl/
 ## Links
 
 - 🤗 HuggingFace Space: [GridMind-RL](https://prajwal782007-gridmind.hf.space)
-- 📝 Blog: [Read the blog post](./HF_BLOG_POST.md)
-- 📓 Colab Notebook: [gridmind_grpo_colab.ipynb](https://colab.research.google.com/github/LO-Kyu/gridmind/blob/main/scripts/gridmind_grpo_colab.ipynb)
-- 🐙 Code Repository: [GitHub](https://github.com/LO-Kyu/gridmind)
+- 📓 Training Notebook: [gridmind_grpo_colab.ipynb](https://colab.research.google.com/github/LO-Kyu/gridmind/blob/main/scripts/gridmind_grpo_colab.ipynb)
+- 📝 Blog Post: [Read the write-up](./HF_BLOG_POST.md)
+- 🐙 GitHub: [Code Repository](https://github.com/LO-Kyu/gridmind)
 
 ---