README.md

metadata

title: ShockMarket OpenEnv
emoji: 🔋
colorFrom: green
colorTo: blue
sdk: docker
app_port: 8000
tags:
  - openenv
base_path: /web

ShockMarket: Agglomeration 2.0 (AG56)

Where Deep Reinforcement Learning Meets Deterministic Optimization

---

OpenEnv Specification Compliant

This repository implements the full OpenEnv interface for a real-world industrial task: Battery Energy Arbitrage.

Environment Description

The agent manages a 20 MWh battery to maximize profit across two simultaneous markets: Energy Arbitrage (buying low, selling high) and Ancillary Services (FCR).

Action Space (Pydantic Model)

• market_choice (float [0.0, 1.0]): Probability/Decision threshold. >=0.6 selects Ancillary, <0.6 selects Arbitrage.
• p_fraction (float [-1.0, 1.0]): Power fraction. Positive means discharge (sell), negative means charge (buy).

Observation Space (Pydantic Model)

• energy_price (float): Current energy market price in /MWh.
• fcr_price (float): Current Frequency Containment Reserve price in /MW.
• soc (float): Current State of Charge (MWh).

Tasks and Graders

The environment provides 3 programmatic tasks with deterministic graders (0.0-1.0 score):

1. Easy: Maximize profit on a stable day with artificially high ancillary service prices.
2. Medium: Arbitrage during high volatility with grid stress.
3. Hard: Maximize revenue under strict terminal SoC constraints and degraded battery efficiency.

Baseline Scores

Running the baseline OpenAI API inference script (backend/baseline.py) with gpt-4o-mini yields the following reproducible scores:

• Easy: 0.85 / 1.00
• Medium: 0.45 / 1.00
• Hard: 0.12 / 1.00

---

The Vision

ShockMarket is a high-performance framework designed to solve the ultimate puzzle in modern energy grids: How do we maximize the value of a Battery Energy Storage System (BESS) across competing markets?

We don't just solve a math problem; we build an intelligent agent that navigates the volatile dance between Energy Arbitrage and Frequency Reserve Markets (FCR & aFRR).

---

System Architecture

The project is structured as a multi-service application to provide a full-stack experience for energy traders and data scientists:

Service	Port	Description
Frontend	3000	React (Vite) interface for strategy visualization.
Backend API	8080	FastAPI server managing RL inference and data orchestration.
Dashboard	8501	Live Streamlit interface for deep-dive technical analytics.

---

Key Features

• Deterministic Co-Optimization: Perfect-foresight day-ahead scheduling using Convex Optimization (CVXPY).
• Multi-Market Synergy: Simultaneous bidding in French Day-Ahead (DA), FCR (Symmetric), and aFRR (UP/DOWN) products.
• Industrial Battery Modeling: Realistic SoC dynamics including efficiency losses (90%) and linear throughput degradation penalties (~15 /MWh).
• Hybrid Intelligence: An advanced Reinforcement Learning (RL) layer (TD3/DDPG) sitting atop the MILP formulation to refine decision-making under uncertainty.

---

The Hybrid Architecture: RL + MILP

While the core of the repository uses Linear Programming (MILP via CVXPY) to find the mathematical optimum for a given set of prices, the Agglomeration 2.0 update introduces a Reinforcement Learning (RL) layer to transform this from a static solver into a dynamic strategy agent.

How RL Enhances the Optimization

The RL agent (implemented via Stable-Baselines3) acts as the "Grand Strategist" atop the "Tactical Solver" (MILP).

1. Adaptive Market Switching

The RL agent learns the probability of market profitability. It makes the high-level decision:

• Should we commit to the Ancillary market (FCR/aFRR) for the capacity payment?
• Or should we keep the battery empty to catch a predicted price spike in the Arbitrage market?

2. Feature-Rich Observation Space

The agent observes Market Signals (DA, FCR, aFRR), Physical State (SoC), and Temporal Context (15-min intervals).

3. The Reward Function: PnL + Sustainability

The agent is trained to maximize PnL while being "nudged" by a Terminal SoC Penalty to return the battery to a target level by midnight.

---

Technical Specifications

Parameter	Specification
Time Grid	15-minute intervals (96 steps/day)
Battery Power	10 MW
Battery Energy	20 MWh
SoC Limits	10% - 90% (Safety Buffer)
Optimization Solver	ECOS (Embedded Conic Solver)
RL Algorithm	TD3 / DDPG (Actor-Critic)

---

Setup & Installation

Option 1: Docker (Recommended for Full Stack)

The easiest way to run the entire project (Frontend, Backend, and Dashboard) in sync.

# 1) Build and start all services
docker-compose up --build

# 2) Access the services:
# - Frontend: http://localhost:3000
# - Backend API: http://localhost:8080
# - Dashboard: http://localhost:8501

Option 2: Local Python Setup (uv optimized)

Best for core RL model development and training.

cd backend

# 1) Create and activate environment
uv venv .venv
source .venv/bin/activate  # Or .venv\Scripts\activate on Windows

# 2) Synchronize dependencies
uv sync

# 3) Run the main training & simulation pipeline
uv run main.py

Option 3: Local Frontend Development

cd frontend
npm install
npm run dev

---

Project Structure

The project follows a clean service-oriented architecture:

Backend

The powerhouse of RL training, data orchestration, and API services.

• src/, rl/, sim/, twin/: Domain logic and optimization modules.
• dashboards/: Technical Streamlit dashboards for live data monitoring.
• data/: Market price data repository.
• models/: Persistent storage for trained RL model weights.
• app.py: FastAPI backend entry point.
• main.py: Entry point for training and simulation.
• Dockerfile: Container configuration for the backend.

Frontend

React-based visualization UI for intuitive strategy monitoring.

• src/: React components and UI logic.
• Dockerfile: Multi-stage build for production-ready nginx serving.
• nginx.conf: Custom routing configuration.

---

Project Context & Credits

This framework is based on industrial-standard battery models and the reference thesis:

"Optimizing Residential Battery Energy Storage Systems Across Frequency Regulation Markets and Energy Arbitrage" Elias Schuhmacher & Eric Rosen (2025).

Developed under the Agglomeration 2.0 - AG56 initiative.