README.md

---
library_name: stable-baselines3
pipeline_tag: reinforcement-learning
tags:
  - reinforcement-learning
  - stable-baselines3
  - ppo
  - gymnasium
  - energy-management
  - solar-microgrid
  - custom-environment
license: apache-2.0
---

# Apollo: Reinforcement Learning Energy Management System

PPO agents trained to manage energy in a simulated solar-powered microgrid — deciding, hour by hour, whether to charge a battery, discharge it, sit idle, or (grid variant) draw from the utility grid. Trained with Stable-Baselines3 inside a custom Gymnasium environment.

This repo hosts two model variants trained on two different environment versions. **The environment code is not included in this repo** — see [Environment & Code](#environment--code) below.

## Model Variants

| Variant | File | Environment | Action Space | Observation Space | Steps Trained |
|---|---|---|---|---|---|
| **Base** | `base/ppo_microgrid_checkpoint_100k.zip` | Standalone (no grid) | 3 discrete: charge / discharge / idle | 4-dim: `[solar, battery, demand, hour]` | 125,000 cumulative |
| **Grid** | `grid/ppo_microgrid_checkpoint_grid_150k.zip` | Grid-enabled, dynamic pricing, stochastic blackouts | 4 discrete: charge / discharge / idle / use grid | 6-dim: `[solar, battery, demand, hour, grid_price, load_shed]` | 220,000 cumulative |

## ⚠️ Important — Evaluation Checkpoint Note

The formal, statistically-evaluated results below (50-seed comparison against Rule-Based and Random baselines) were measured on **earlier checkpoints** than the ones hosted in this repo:

- Base results correspond to `ppo_microgrid_checkpoint_20k` (25,000 cumulative steps)
- Grid results correspond to `ppo_microgrid_checkpoint_grid_60k` (70,000 cumulative steps)

The checkpoints uploaded here (`_100k` / `_grid_150k`) were trained **further** after that evaluation but have **not been independently re-run through the full 50-seed evaluation protocol.** Training diagnostics observed diminishing returns beyond ~50K additional steps (only ~7-8% further improvement), so performance is expected to be comparable or modestly better — but this has not been formally confirmed for these exact checkpoints. If you need numbers that are guaranteed to match a specific file 1:1, use the `_20k` / `_grid_60k` checkpoints from the [GitHub repo](https://github.com/ali591195/apollo-rl-ems) instead.

## Evaluation Results (measured on `_20k` / `_grid_60k` — see note above)

### Base Environment
| Policy | Avg Reward | Avg Unmet Demand | Avg Waste | Avg Penalized Waste | Reward Std Dev |
|---|---|---|---|---|---|
| **PPO** | **-10.709** | 2.118 | 2.096 | 0.001 | **1.724** |
| Random | -15.930 | 2.340 | 2.204 | 1.744 | 3.916 |
| Rule-Based | -23.284 | **1.769** | 1.872 | 0.000 | 4.107 |

### Grid-Enabled Environment
| Policy | Avg Reward | Avg Unmet Demand | Avg Waste | Avg Penalized Waste | Reward Std Dev |
|---|---|---|---|---|---|
| **PPO** | **-3.831** | **0.407** | 2.576 | 0.109 | **2.251** |
| Rule-Based | -4.460 | 0.488 | **2.297** | **0.000** | 3.157 |
| Random | -14.203 | 2.069 | 2.801 | 0.487 | 3.711 |

PPO outperforms Rule-Based on reward by ~54% and Random by ~33% in the base environment, and wins on both reward and unmet demand in the harder grid-enabled environment, while remaining the most stable policy (lowest reward variance) in both.

## Environment & Code

This model depends on a custom `MicrogridEnv` (Gymnasium) that is **not bundled in this repo**. Get it from the source GitHub repository:

**[github.com/ali591195/apollo-rl-ems](https://github.com/ali591195/apollo-rl-ems)**

That repo contains the full environment (`src/env.py`, `src/reward.py`, `src/simulation/`), the original training notebook, and the complete write-up this model card summarizes.

## How to Use

```bash
git clone https://github.com/ali591195/apollo-rl-ems
cd apollo-rl-ems
pip install -r requirements.txt
```

```python
from huggingface_hub import hf_hub_download
from stable_baselines3 import PPO
from src.env import MicrogridEnv
from src.reward import compute_reward

# Download the desired checkpoint from this repo
checkpoint_path = hf_hub_download(
    repo_id="ali591195/apollo-rl-ems",   # update to your actual HF repo id
    filename="grid/ppo_microgrid_checkpoint_grid_150k.zip"
)

# grid_mode must match the variant you downloaded
env = MicrogridEnv(compute_reward, grid_mode=True)
model = PPO.load(checkpoint_path, env=env)

obs, _ = env.reset()
for _ in range(24):  # one simulated day
    action, _ = model.predict(obs)
    obs, reward, done, truncated, info = env.step(action)
```

> Match `grid_mode` to the variant you load: `False` for `base/`, `True` for `grid/`. Base and grid checkpoints were trained on different observation/action space shapes and are not interchangeable.

## Training Details

- **Algorithm:** PPO (Stable-Baselines3 2.9.0)
- **Policy:** MLP, discrete action space
- **Training schedule:** staged → diagnostic (2-5K steps) → early validation (10-20K) → extended (50K+), checkpointed after each phase
- **Base environment:** 5K → 20K → 100K step phases (125,000 cumulative)
- **Grid environment:** 10K → 60K → 150K step phases (220,000 cumulative)

## Reward Function

```
reward = -(5.0 × unmet_demand)
         -(0.2 × wasted_energy, only if battery had room to store it)
         -(2.0 × battery_misuse, running battery near-empty before low-solar periods)
         -(1.5 × grid_price, only if grid action used and grid is online)
```

## Limitations

- The learned policy depends on the chosen reward function. Although the reward weights were iteratively refined using domain reasoning and empirical evaluation, alternative formulations or weightings may produce different optimization strategies.
- In the base environment, PPO accepts ~20% more unmet demand than the Rule-Based baseline in exchange for better overall reward, a real trade-off, not a strict win on every axis.
- Solar generation, demand, and grid pricing are synthetic models, not real historical weather/tariff data.
- No temperature effects modeled.
- See the [GitHub repo](https://github.com/ali591195/apollo-rl-ems) for the full limitations and future-work discussion (a storage-scaling and action-awareness upgrade is in progress).

## License

Apache-2.0

## Author

Ali Hassan — ML Developer/Engineer, Lahore, Pakistan · [LinkedIn](https://www.linkedin.com/in/ali-hassan-483977245/) · [GitHub](https://github.com/ali591195) · [Hugging Face](https://huggingface.co/ali591195)