feat: add GridMind-RL inference script and update documentation

README.md

@@ -4,13 +4,84 @@ GridMind-RL is an OpenEnv-compliant reinforcement learning environment simulatin
 
 An RL agent acts as the energy controller, shaping electrical load profiles by adjusting HVAC setpoints, managing thermal storage, and scheduling batch processes. The goal is to optimize operations in response to real-time electricity prices, grid carbon intensity, and utility demand-response signals.
 
-## Architecture
+---
+
+## 🙋 Beginner? Start Here
+
+If you're new to this project, you probably have these questions:
+
+### ❓ Why do I need an API?
+
+In this project, the "brain" that makes energy decisions is an **AI language model (LLM)** — like Llama.
+
+Instead of running the full AI model on your own computer (which requires a powerful GPU), you connect to an **API** (Application Programming Interface) — a remote server that already has the model running. You send it the current building state (temperature, price, etc.) and it sends back what action to take (e.g. "charge thermal storage").
+
+Think of it like this:
+```
+Your Computer  ──(asks question)──►  API Server (has the AI)  ──(sends answer)──►  Your Computer
+```
+
+Without an API key, your script has no way to reach the AI model and the inference won't work.
+
+---
+
+### ❓ How do I get an API key?
+
+This project uses **Hugging Face** — a free platform that hosts AI models.
+
+#### Step-by-step:
+
+1. **Create a free account** at [https://huggingface.co/join](https://huggingface.co/join)
+
+2. **Go to your profile → Settings → Access Tokens**
+   Direct link: [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)
+
+3. Click **"New token"**, give it any name (e.g. `gridmind`), and select role **"Read"**
+
+4. Copy the token — it looks like: `hf_aBcDeFgHiJkLmNoPqRsTuVwXyZ`
+
+5. You'll paste this token in the terminal when running the project (shown below)
+
+> **💡 It's free!** Hugging Face's inference API has a free tier that's enough to run this project.
+
+---
+
+### ❓ Why Llama? What even is Llama?
+
+**Llama** (Large Language Model Meta AI) is an open-source AI model made by Meta (Facebook). Think of it like a smarter, programmable version of ChatGPT that you can use via an API.
+
+**Why this project uses Llama specifically:**
+
+| Reason | Explanation |
+|--------|-------------|
+| 🆓 Free to use | Available on Hugging Face at no cost |
+| 📖 Open-source | The weights and code are public — no black box |
+| 🧠 Smart enough | Llama 3.1 8B is capable of reading sensor data and outputting valid JSON actions |
+| ⚡ Fast | The 8B (8 billion parameter) version is small enough to run quickly on Hugging Face's servers |
+| 🔄 OpenAI-compatible | It uses the same API format as OpenAI, so the code works with many models |
+
+The model reads the building state (temperature, electricity price, grid stress) and outputs a JSON action like:
+```json
+{
+  "hvac_power_level": 0.4,
+  "thermal_charge_rate": 0.5,
+  "batch_job_slot": 2,
+  "load_shed_fraction": 0.0,
+  "building_id": 0
+}
+```
+
+> **You can also swap Llama for any other OpenAI-compatible model** (GPT-4, Mistral, etc.) by changing the environment variables.
+
+---
+
+## 🏗️ Architecture
 
 ```text
  ┌──────────────────────┐        ┌─────────────────────────────┐
  │                      │        │                             │
  │    LLM RL Agent      │◄───────┤    GridMind-RL Server       │
- │   (Inference Script) │ POST   │    (Go OpenEnv Backend)     │
+ │   (Python Script)    │ POST   │    (Go OpenEnv Backend)     │
  │                      ├───────►│  Port 7860                  │
  └──────────────────────┘ Action │                             │
                                  └──────────────┬──────────────┘
@@ -24,7 +95,162 @@ An RL agent acts as the energy controller, shaping electrical load profiles by a
                                  └─────────────────────────────┘
 ```
 
-## Observation Space
+---
+
+## 🚀 How to Run the Project (Step by Step)
+
+There are **two ways** to run this project:
+- **Option A** — Using Docker (recommended, easiest)
+- **Option B** — Running manually without Docker
+
+---
+
+### Option A: Docker (Recommended)
+
+Docker packages everything into a container so you don't need to install Go, Python versions, etc. separately.
+
+#### Prerequisites
+
+- Install Docker Desktop: [https://www.docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop)
+- A Hugging Face API token (see above ☝️)
+
+#### Step 1 — Build the Docker image
+
+Open a terminal in the project folder and run:
+
+```bash
+docker build -t gridmind-rl .
+```
+
+This may take a few minutes the first time.
+
+#### Step 2 — Start the environment server
+
+```bash
+docker run -p 7860:7860 -p 7861:7861 gridmind-rl
+```
+
+You should see the server start. Keep this terminal open.
+
+- **Environment API:** http://localhost:7860
+- **Visualization Dashboard:** http://localhost:7861
+
+#### Step 3 — Install Python dependencies
+
+Open a **new terminal** (keep the Docker one running) and run:
+
+```bash
+pip install -r python/requirements.txt
+```
+
+#### Step 4 — Set your API credentials
+
+**On Windows (Command Prompt):**
+```cmd
+set API_BASE_URL=https://router.huggingface.co/v1
+set MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
+set HF_TOKEN=hf_your_token_here
+```
+
+**On Windows (PowerShell):**
+```powershell
+$env:API_BASE_URL = "https://router.huggingface.co/v1"
+$env:MODEL_NAME   = "meta-llama/Llama-3.1-8B-Instruct"
+$env:HF_TOKEN     = "hf_your_token_here"
+```
+
+**On Mac/Linux:**
+```bash
+export API_BASE_URL=https://router.huggingface.co/v1
+export MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
+export HF_TOKEN=hf_your_token_here
+```
+
+Replace `hf_your_token_here` with your actual Hugging Face token.
+
+#### Step 5 — Run the AI agent
+
+```bash
+python python/inference.py --episodes 3
+```
+
+You'll see the agent play through 3 episodes across all 3 tasks and print scores.
+
+---
+
+### Option B: Manual (Without Docker)
+
+Use this if you don't have Docker installed.
+
+#### Prerequisites
+
+- [Go 1.21+](https://go.dev/dl/) — for running the environment server
+- [Python 3.9+](https://www.python.org/downloads/) — for the AI agent script
+- A Hugging Face API token (see above ☝️)
+
+#### Step 1 — Start the Go environment server
+
+```bash
+go run main.go
+```
+
+The server starts on port `7860`. Keep this terminal open.
+
+#### Step 2 — Open a new terminal and install Python dependencies
+
+```bash
+pip install -r python/requirements.txt
+```
+
+#### Step 3 — Set your API credentials (same as Option A, Step 4 above)
+
+#### Step 4 — Validate the environment is working
+
+```bash
+python python/validate.py --env-url http://localhost:7860
+```
+
+You should see a series of checks pass. If they do, you're good to go.
+
+#### Step 5 — Run the AI agent
+
+```bash
+python python/inference.py --episodes 3
+```
+
+---
+
+## 📊 What Happens When You Run It
+
+The agent runs through **3 tasks** (Easy → Medium → Hard), each for the number of episodes you specify:
+
+| Task | Difficulty | Goal |
+|------|-----------|------|
+| Task 1 | Easy | Minimize energy costs only |
+| Task 2 | Medium | Minimize costs + keep temperature 19°C–23°C |
+| Task 3 | Hard | Costs + temperature + batch job deadlines + grid stress response |
+
+At the end, you'll see a score table like:
+```
+============================================================
+BASELINE SCORES SUMMARY
+============================================================
+Task       Model                          Score      Episodes
+------------------------------------------------------------
+Task 1     meta-llama/Llama-3.1-8B-Instruct  0.7823     3
+Task 2     meta-llama/Llama-3.1-8B-Instruct  0.6541     3
+Task 3     meta-llama/Llama-3.1-8B-Instruct  0.5102     3
+------------------------------------------------------------
+Overall                                    0.6489
+```
+
+Results are also saved to `baseline_scores.json`.
+
+---
+
+## 📐 Observation Space
+
+These are the sensor readings the agent sees at each step:
 
 | Name | Type | Range | Description |
 |------|------|-------|-------------|
@@ -40,7 +266,11 @@ An RL agent acts as the energy controller, shaping electrical load profiles by a
 | `step` | int | [0, 95] | Current episode timestep (15-min intervals over 24h). |
 | `building_id` | int | [0, 2] | ID of the building in multi-building federated mode. |
 
-## Action Space
+---
+
+## 🕹️ Action Space
+
+These are the controls the agent outputs at each step:
 
 | Name | Type | Range | Description |
 |------|------|-------|-------------|
@@ -50,18 +280,9 @@ An RL agent acts as the energy controller, shaping electrical load profiles by a
 | `load_shed_fraction` | float | [0.0, 0.5] | Fraction of non-critical load to shed (max 50%). |
 | `building_id` | int | [0, 2] | Select which building to apply this action to (federation). |
 
-## Tasks
-
-GridMind-RL features 3 progressively difficult tasks:
+---
 
-1. **Task 1: Cost Minimization (Easy)**
-   Minimize total energy costs by moving load to off-peak periods using thermal storage. No temperature constraints.
-2. **Task 2: Temperature Management (Medium)**
-   Minimize costs while keeping indoor temperatures strictly within 19°C – 23°C.
-3. **Task 3: Full Demand Response (Hard)**
-   Minimize cost, maintain temperature, successfully schedule batch jobs before deadlines, and shed loads when the grid stress signal exceeds 0.7.
-
-## Reward Function
+## 🏆 Reward Function
 
 The dense reward includes several components:
 * **Cost Savings:** Proportional to energy savings vs the baseline flat tariff policy.
@@ -73,38 +294,22 @@ The dense reward includes several components:
 
 *Exploit Detection:* The grader detects degenerate strategies (e.g. permanently shedding 40% load) and applies up to a 30% score penalty.
 
-## Usage
-
-### Local Docker Build
+---
 
-```bash
-docker build -t gridmind-rl .
-docker run -p 7860:7860 -p 7861:7861 gridmind-rl
-```
+## 🔧 Extensions
 
-* Backend OpenEnv server: http://localhost:7860
-* Visualization Dashboard: http://localhost:7861
-
-### Validating the Environment
-
-```bash
-python python/validate.py --env-url http://localhost:7860
-```
-
-### Running Baseline Inference
-
-```bash
-export API_BASE_URL=https://api-inference.huggingface.co/v1
-export MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
-export HF_TOKEN=your_token
+* **Multi-building mode:** Switch the environment to 3 buildings via `POST /reset {"num_buildings": 3}` and output action arrays for coordinated dispatch.
+* **Use a different model:** Just change `MODEL_NAME` to any OpenAI-compatible model (e.g. `mistralai/Mistral-7B-Instruct-v0.3`).
+* **Add new tasks:** Edit `env/tasks.go` and implement a new `gradeTaskX` component.
 
-# Install dependencies
-pip install -r python/requirements.txt
+---
 
-# Run inference
-python python/inference.py --episodes 3
-```
+## ❓ Troubleshooting
 
-## Extensions
-* **Multi-building mode:** Switch the environment to 3 buildings via `POST /reset {"num_buildings": 3}` and output action arrays for coordinated dispatch.
-* **Add new tasks:** Edit `env/tasks.go` and implement a new `gradeTaskX` component.
+| Problem | Fix |
+|---------|-----|
+| `Connection refused` on port 7860 | Make sure the Docker container or `go run main.go` is still running |
+| `401 Unauthorized` from Hugging Face | Your `HF_TOKEN` is wrong or expired — generate a new one |
+| `Model not found` error | Some large models require you to accept terms on Hugging Face first. Go to the model page and click "Agree to terms" |
+| Python package errors | Make sure you ran `pip install -r python/requirements.txt` |
+| `docker: command not found` | Install Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop) |

python/inference.py

@@ -5,7 +5,7 @@ Runs an LLM agent against all 3 tasks for N episodes each.
 Uses OpenAI-compatible API via API_BASE_URL / MODEL_NAME / HF_TOKEN environment variables.
 
 Usage:
-    export API_BASE_URL=https://api-inference.huggingface.co/v1
+    export API_BASE_URL=https://router.huggingface.co/v1
     export MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
     export HF_TOKEN=hf_xxxx
     python python/inference.py [--episodes 3] [--env-url http://localhost:7860]
@@ -26,7 +26,7 @@ from openai import OpenAI
 # ── Constants ──────────────────────────────────────────────────────────────
 
 ENV_URL = os.getenv("ENV_URL", "http://localhost:7860")
-API_BASE_URL = os.getenv("API_BASE_URL", "https://api-inference.huggingface.co/v1")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
 MODEL_NAME = os.getenv("MODEL_NAME", "meta-llama/Llama-3.1-8B-Instruct")
 HF_TOKEN = os.getenv("HF_TOKEN", "")
 DEFAULT_EPISODES = 3
@@ -245,6 +245,11 @@ def run_episode(env_client: GridMindEnvClient, agent: LLMAgent,
         action = agent.choose_action(obs, task_id)
         step_resp = env_client.step(action)
 
+        if step_resp is None or "observation" not in step_resp:
+            print(f"  [WARN] step {_step}: server returned invalid response, skipping step")
+            _step += 1
+            break
+
         obs = step_resp["observation"]
         total_reward += step_resp["reward"]
         total_steps += 1