README.md

---
title: EnergyStock BESS-RL
emoji: ⚡
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: false
---

# ⚡ BESS-RL: Battery Energy Storage System Reinforcement Learning

[![HF Space](https://img.shields.io/badge/🤗%20Space-PowerGrid-blue)](https://huggingface.co/spaces/saiteja020/PowerGrid)
[![OpenEnv](https://img.shields.io/badge/OpenEnv-Compliant-green)](https://openenv.ai)

An industrial-grade, **OpenEnv-compliant** reinforcement learning environment for optimizing **Battery Energy Storage System (BESS)** dispatch. Develop agents that manage grid-scale batteries to maximize revenue across multiple simultaneous streams using real-world PJM electricity market data.

---

## 🌍 Environment Overiew

The BESS-RL environment simulates the hourly operation of a battery connected to the PJM (Pennsylvania-New Jersey-Maryland Interconnection) grid. Agents must solve a complex co-optimization problem involving three distinct revenue and reliability objectives:

1.  **Energy Arbitrage (EA):** Capitalizing on price volatility by charging during low-price periods and discharging when prices peak.
2.  **Frequency Regulation (FR):** Providing grid stability by following the PJM RegD (Dynamic Regulation) signal in real-time.
3.  **Peak Shaving (PS):** Managing local grid stress by ensuring net load remains below critical thresholds to avoid heavy demand charges.

### Key Features
- **Real Data:** Driven by authentic PJM hourly Locational Marginal Prices (LMP), RegD signals, and load profiles.
- **Physics-Informed:** Includes battery degradation (cycle costs), efficiency losses, and SOC (State of Charge) constraints.
- **Task Tiers:** Built-in curriculum from single-objective arbitrage to full multi-objective co-optimization.

---

## 📊 Observation Space

The agent receives a **6-dimensional** vector at each timestep representing the current state of the market and the battery.

| Index | Field | Range | Units | Description |
| :--- | :--- | :--- | :--- | :--- |
| 0 | `hour_of_day` | 0 – 23 | Hours | Current hour (normalized to float) |
| 1 | `soc` | 0.0 – 1.0 | % | Battery State of Charge (0% to 100%) |
| 2 | `price_lmp` | 0 – 200+ | $/MWh | Current Locational Marginal Price |
| 3 | `p_avg` | 0 – 200+ | $/MWh | 24-hour rolling average LMP |
| 4 | `freq_regd` | -1.0 – 1.0 | Signal | PJM RegD frequency regulation signal |
| 5 | `load_mw` | 0 – 50+ | MW | Local grid load demand |

---

## 🕹️ Action Space

The action space is a **continuous** vector of 3 values, each bounded in `[-1.0, 1.0]`.

| Index | Component | Range | Physical Meaning |
| :--- | :--- | :--- | :--- |
| 0 | `a_PS` | `[-1, 1]` | Peak Shaving dispatch request |
| 1 | `a_EA` | `[-1, 1]` | Energy Arbitrage dispatch request |
| 2 | `a_FR` | `[-1, 1]` | Frequency Regulation dispatch request |

> [!NOTE]
> The environment implements a "clipping aggregator": `a_final = clip(a_PS + a_EA + a_FR, -1.0, 1.0)`.
> `+1.0` represents Max Charge Power, while `-1.0` represents Max Discharge Power.

---

## 🏆 Multi-Objective Tasks

| Task | Level | Objectives | Learning Focus |
| :--- | :--- | :--- | :--- |
| `easy` | Beginner | `EA` | Price forecasting and temporal arbitrage timing. |
| `medium` | Intermediate | `EA` + `FR` | Balancing high-frequency signal tracking with market trends. |
| `hard` | Advanced | `EA` + `FR` + `PS` | Multi-constraint optimization under grid stress. |

---

## 🛠️ Setup & Installation

### Option 1: Docker (Recommended)
The project is containerized for seamless deployment.

```bash
# Build and start both the backend server and React dashboard
docker-compose up --build
```
The application will be available at [http://localhost:7860](http://localhost:7860).

### Option 2: Manual Setup
If you prefer to run the components locally for development:

```bash
# 1. Install dependencies
pip install -r backend/requirements.txt
pip install openai torch numpy pandas pydantic python-dotenv

# 2. Configure environment
cp .env.example .env
# Open .env and add your HF_TOKEN and other API keys

# 3. Start the FastAPI server
python backend/main.py
```

---

## 🚀 Usage

### Running the Evaluation Harness
The `inference.py` script runs a full evaluation of the SAC agent across all task tiers and outputs OpenEnv-compliant logs.

```bash
# Requires HF_TOKEN, MODEL_NAME, and API_BASE_URL env variables
python inference.py
```

### Training Your Own Agent
We use **Soft Actor-Critic (SAC)** to train the dispatch models.

```bash
# Train on the hard task for 500 episodes
python train/trainer.py --task hard --episodes 500
```

### AI-Powered Evaluation (llm-evaluator)
Run the AI-powered evaluation to get a qualitative verdict and a robust performance score from the LLM evaluator.

```bash
# Evaluate specific task tiers
python train/llm_evaluate.py --task easy
python train/llm_evaluate.py --task medium
python train/llm_evaluate.py --task hard
```

---

## 🛠️ Environment Configuration

| Variable | Description |
| :--- | :--- |
| `API_BASE_URL` | Endpoint for the LLM advisory layer (OpenAI compatible) |
| `MODEL_NAME` | The model identifier for inference (e.g., `meta-llama/Llama-3.1-8B-Instruct`) |
| `HF_TOKEN` | Your Hugging Face API key for inference and model syncing |

---

## ✅ Submission Validation

Before submitting your project, you can use the `validate-submission.sh` script to ensure everything is configured correctly.

### What it validates:
1.  **Connectivity**: Checks if your HuggingFace Space is live and the `/reset` endpoint is responsive.
2.  **Containerization**: Verifies that your `Dockerfile` builds successfully (typically within a 10-minute timeout).
3.  **OpenEnv Compliance**: Runs `openenv validate` to ensure your repository structure and environment meet the OpenEnv standards.

### How to run:

You can run the script locally or directly via `curl`. 

**Prerequisites:**
- **Docker**: [Install Docker](https://docs.docker.com/get-docker/)
- **OpenEnv**: `pip install openenv-core`

**Run locally:**
```bash
chmod +x validate-submission.sh
./validate-submission.sh <your_hf_space_url>
```

**Run via curl:**
```bash
curl -fsSL https://raw.githubusercontent.com/PuttaVenkataSai/PowerGrid/main/validate-submission.sh | bash -s -- <your_hf_space_url>
```

*Example:* `./validate-submission.sh https://saiteja020-powergrid.hf.space`

---

## 🏗️ Architecture

- **Agent:** SAC with Twin Critics and automatic entropy tuning for robust policy learning.
- **Frontend:** React + Tailwind CSS dashboard for real-time visualization of BESS dispatch.
- **Backend:** FastAPI high-performance server exposing OpenEnv standard endpoints (`/reset`, `/step`).
- **Validation:** Internal `validate-submission.sh` script to ensure HF Space and OpenEnv compliance.