docs/task_4.md

Task 4 — multi_stage_cascade

Purpose

Task 4 is the hardest benchmark task. Three lines are disconnected, load is increased, and the task is designed around guaranteed multi-stage degradation rather than full prevention. The goal is to preserve as much load as possible across stage boundaries over 30 steps.

Current task definition in code:

• description and scenario setup: server/tasks.py
• reward: server/environment.py
• grader: server/graders.py

What We Changed

This task changed the most during development.

Main changes:

• made it explicitly stage-aware in both prompt and reward logic
• added stage boundary assessment and island-availability tracking
• added controlled-islanding and redispatch guidance to the prompt
• changed the reward to penalize load loss at stage boundaries instead of only rewarding short-term survival
• changed the grader to score stage completion, load preservation, island quality, and stage-wise stabilization speed

These changes were motivated by multi-stage cascading-failure literature, where greedy single-stage action selection is known to miss later-stage consequences. [1][2]

Current Implementation

Reset:

• three lines disconnected
• load increase around the expert benchmark setting
• overflow window tightened for faster propagation
• 30 steps split into three conceptual stages

Prompt-side task rules now include:

• explicit stage context such as stage_1_of_3
• steps remaining to the next boundary
• available_load_ratio
• available_island_ratio
• guidance to avoid stranding load in islands without enough generation
• separate controlled-islanding and redispatch guidance blocks

Current reward shape:

• generation cost penalty
• positive term for available-island convergence
• load-loss penalty at stage boundaries
• heavy penalties for blackout or convergence failure
• positive win reward at the horizon if enough load is preserved

Current grader:

• 30% stage completion
• 40% load preservation
• 20% island quality
• 10% speed of stabilization within each stage

What We Observed

This is the task where the SFT model most clearly outperformed the base model while staying safe.

Final SFT result, seed block 0..4:

• 0.9156444

Final SFT result, unseen seeds 100..102:

• 0.9069863

Base-model result on seed block 0..4:

• 0.000

This task also exposed the limit of the original offline GRPO setup:

• completed GRPO runs preserved the SFT behavior
• they did not improve the multi_stage_cascade score

Current Limitation

The task is intentionally constrained by verified candidate actions and a fixed benchmark horizon. That makes it robust and reproducible, but it also means some genuine long-horizon improvements may be hard to expose unless the candidate set itself contains them.

References

[1] Mitigating Multi-Stage Cascading Failure by Reinforcement Learning (ISGT Asia 2019):
https://vbn.aau.dk/en/publications/mitigating-multi-stage-cascading-failure-by-reinforcement-learnin/

[2] Deep Reinforcement Learning for Power Grid Multi-Stage Cascading Failure Mitigation:
https://www.climatechange.ai/papers/iclr2025/1

[3] Local implementation:
server/tasks.py, server/environment.py, server/graders.py, inference.py