| ---
|
| title: ShockMarket OpenEnv
|
| emoji: 🔋
|
| colorFrom: green
|
| colorTo: blue
|
| sdk: docker
|
| app_port: 8000
|
| tags:
|
| - openenv
|
| base_path: /web
|
| ---
|
|
|
| # ShockMarket: Agglomeration 2.0 (AG56)
|
|
|
| ### *Where Deep Reinforcement Learning Meets Deterministic Optimization*
|
|
|
| ---
|
|
|
| ## OpenEnv Specification Compliant
|
| This repository implements the full **OpenEnv** interface for a real-world industrial task: Battery Energy Arbitrage.
|
|
|
| ### Environment Description
|
| The agent manages a 20 MWh battery to maximize profit across two simultaneous markets: Energy Arbitrage (buying low, selling high) and Ancillary Services (FCR).
|
|
|
| ### Action Space (Pydantic Model)
|
| - `market_choice` (float [0.0, 1.0]): Probability/Decision threshold. `>=0.6` selects Ancillary, `<0.6` selects Arbitrage.
|
| - `p_fraction` (float [-1.0, 1.0]): Power fraction. Positive means discharge (sell), negative means charge (buy).
|
|
|
| ### Observation Space (Pydantic Model)
|
| - `energy_price` (float): Current energy market price in /MWh.
|
| - `fcr_price` (float): Current Frequency Containment Reserve price in /MW.
|
| - `soc` (float): Current State of Charge (MWh).
|
|
|
| ### Tasks and Graders
|
| The environment provides 3 programmatic tasks with deterministic graders (0.0-1.0 score):
|
| 1. **Easy**: Maximize profit on a stable day with artificially high ancillary service prices.
|
| 2. **Medium**: Arbitrage during high volatility with grid stress.
|
| 3. **Hard**: Maximize revenue under strict terminal SoC constraints and degraded battery efficiency.
|
|
|
| ### Baseline Scores
|
| Running the baseline OpenAI API inference script (`backend/baseline.py`) with `gpt-4o-mini` yields the following reproducible scores:
|
| - **Easy**: 0.85 / 1.00
|
| - **Medium**: 0.45 / 1.00
|
| - **Hard**: 0.12 / 1.00
|
|
|
| ---
|
|
|
| ## The Vision
|
|
|
| **ShockMarket** is a high-performance framework designed to solve the ultimate puzzle in modern energy grids: **How do we maximize the value of a Battery Energy Storage System (BESS) across competing markets?**
|
|
|
| We don't just solve a math problem; we build an intelligent agent that navigates the volatile dance between **Energy Arbitrage** and **Frequency Reserve Markets (FCR & aFRR)**.
|
|
|
| ---
|
|
|
| ## System Architecture
|
|
|
| The project is structured as a multi-service application to provide a full-stack experience for energy traders and data scientists:
|
|
|
| | Service | Port | Description |
|
| | :--- | :--- | :--- |
|
| | **Frontend** | `3000` | React (Vite) interface for strategy visualization. |
|
| | **Backend API** | `8080` | FastAPI server managing RL inference and data orchestration. |
|
| | **Dashboard** | `8501` | Live Streamlit interface for deep-dive technical analytics. |
|
|
|
| ---
|
|
|
| ## Key Features
|
|
|
| * **Deterministic Co-Optimization:** Perfect-foresight day-ahead scheduling using Convex Optimization (CVXPY).
|
| * **Multi-Market Synergy:** Simultaneous bidding in French Day-Ahead (DA), FCR (Symmetric), and aFRR (UP/DOWN) products.
|
| * **Industrial Battery Modeling:** Realistic SoC dynamics including efficiency losses (90%) and linear throughput degradation penalties (~15 /MWh).
|
| * **Hybrid Intelligence:** An advanced **Reinforcement Learning (RL)** layer (TD3/DDPG) sitting atop the MILP formulation to refine decision-making under uncertainty.
|
|
|
| ---
|
|
|
| ## The Hybrid Architecture: RL + MILP
|
|
|
| While the core of the repository uses **Linear Programming (MILP via CVXPY)** to find the mathematical optimum for a given set of prices, the **Agglomeration 2.0** update introduces a **Reinforcement Learning (RL)** layer to transform this from a static solver into a dynamic strategy agent.
|
|
|
| ### How RL Enhances the Optimization
|
|
|
| The RL agent (implemented via `Stable-Baselines3`) acts as the "Grand Strategist" atop the "Tactical Solver" (MILP).
|
|
|
| #### 1. Adaptive Market Switching
|
|
|
| The RL agent learns the **probability of market profitability**. It makes the high-level decision:
|
| * *Should we commit to the Ancillary market (FCR/aFRR) for the capacity payment?*
|
| * *Or should we keep the battery empty to catch a predicted price spike in the Arbitrage market?*
|
|
|
| #### 2. Feature-Rich Observation Space
|
| The agent observes Market Signals (DA, FCR, aFRR), Physical State (SoC), and Temporal Context (15-min intervals).
|
|
|
| #### 3. The Reward Function: PnL + Sustainability
|
| The agent is trained to maximize PnL while being "nudged" by a **Terminal SoC Penalty** to return the battery to a target level by midnight.
|
|
|
| ---
|
|
|
| ## Technical Specifications
|
|
|
| | Parameter | Specification |
|
| | --- | --- |
|
| | **Time Grid** | 15-minute intervals (96 steps/day) |
|
| | **Battery Power** | 10 MW |
|
| | **Battery Energy** | 20 MWh |
|
| | **SoC Limits** | 10% - 90% (Safety Buffer) |
|
| | **Optimization Solver** | ECOS (Embedded Conic Solver) |
|
| | **RL Algorithm** | TD3 / DDPG (Actor-Critic) |
|
|
|
| ---
|
|
|
| ## Setup & Installation
|
|
|
| ### Option 1: Docker (Recommended for Full Stack)
|
| The easiest way to run the entire project (Frontend, Backend, and Dashboard) in sync.
|
|
|
| ```bash
|
| # 1) Build and start all services
|
| docker-compose up --build
|
|
|
| # 2) Access the services:
|
| # - Frontend: http://localhost:3000
|
| # - Backend API: http://localhost:8080
|
| # - Dashboard: http://localhost:8501
|
| ```
|
|
|
| ### Option 2: Local Python Setup (`uv` optimized)
|
| Best for core RL model development and training.
|
|
|
| ```bash
|
| cd backend
|
|
|
| # 1) Create and activate environment
|
| uv venv .venv
|
| source .venv/bin/activate # Or .venv\Scripts\activate on Windows
|
|
|
| # 2) Synchronize dependencies
|
| uv sync
|
|
|
| # 3) Run the main training & simulation pipeline
|
| uv run main.py
|
| ```
|
|
|
| ### Option 3: Local Frontend Development
|
| ```bash
|
| cd frontend
|
| npm install
|
| npm run dev
|
| ```
|
|
|
| ---
|
|
|
| ## Project Structure
|
|
|
| The project follows a clean service-oriented architecture:
|
|
|
| ### [Backend](file:///backend)
|
| The powerhouse of RL training, data orchestration, and API services.
|
| - `src/`, `rl/`, `sim/`, `twin/`: Domain logic and optimization modules.
|
| - `dashboards/`: Technical Streamlit dashboards for live data monitoring.
|
| - `data/`: Market price data repository.
|
| - `models/`: Persistent storage for trained RL model weights.
|
| - `app.py`: FastAPI backend entry point.
|
| - `main.py`: Entry point for training and simulation.
|
| - `Dockerfile`: Container configuration for the backend.
|
|
|
| ### [Frontend](file:///frontend)
|
| React-based visualization UI for intuitive strategy monitoring.
|
| - `src/`: React components and UI logic.
|
| - `Dockerfile`: Multi-stage build for production-ready nginx serving.
|
| - `nginx.conf`: Custom routing configuration.
|
|
|
| ---
|
|
|
| ## Project Context & Credits
|
|
|
| This framework is based on industrial-standard battery models and the reference thesis:
|
|
|
| > *"Optimizing Residential Battery Energy Storage Systems Across Frequency Regulation Markets and Energy Arbitrage"* **Elias Schuhmacher & Eric Rosen (2025)**.
|
|
|
| Developed under the **Agglomeration 2.0 - AG56** initiative.
|
|
|
