Spaces:

---

gridfm

/

genco-challenge

Like 24

Running

Trust, but verify: making GENCO safe for N-k contingency screening

Participant: Alireza Shooshtari, Institut de Recerca en Energia de Catalunya (IREC) · Contact: a.shooshtari1996@gmail.com

Code and results: github.com/Alirezashooshtari/genco-trust-but-verify · Fine-tuned model and warning: Alirezashooshtari/genco-texas-clustered · Case explorer: genco-contingency-verifier

In brief. A contingency screen is judged by its worst miss, not its average error. On outages clustered in one area, the kind a storm causes, GENCO fails quietly: it returns a complete state even where no Newton start converges, with nothing to mark the case. We trace this to a measurable flaw in how it responds to system-wide stress, fix that flaw inside the model, and wrap GENCO in a check that never passes a case it cannot verify.

1. The problem: the cases that matter are the ones GENCO has seen least

GENCO answers a contingency in one forward pass, so it is a natural engine for screening thousands of N-k cases. But a screen earns its keep on the rare, severe cases: a corridor cut by a storm, a wildfire taking out lines in one region, a grid pushed close to voltage collapse. These outages are spatially clustered, unlike the random outages a model is usually tested on. An operator using GENCO needs two answers: when can its output be trusted, and what happens when it cannot?

We study both on the released Texas 2,000-bus grid and checkpoint (gridfm/genco-pf-contingency), with gridfm-graphkit, and repeat the key tests on IEEE 118 with the model the authors released for that grid. Figure 1 shows where we end up.

Figure 1. Every contingency ends in one of three outcomes. GENCO alone has no third outcome: it returns a full state for every case.

2. Where GENCO fails: quietly, on clustered outages

We built clustered outages by removing lines within one area of the grid, and compared them with random outages of the same size. GENCO's median worst-bus voltage error rises from about 0.002 p.u. on random outages to 0.008 p.u. at N-10 and 0.042 p.u. at N-20 when the outages cluster: 4.0 and 16.5 times higher.

The more serious failure is what GENCO does when there is no answer to give. For 33 of 120 clustered N-20 storms, no Newton start converges at the actual load. Continuation places the fold of the AC solution below the actual load for 32 of them; for the last, no start converged anywhere in the tested load range. GENCO returns a complete state for all 33. Nothing in that state sets these storms apart: at the plain limits it shows violations in 30 of the 33, but also in 79 of the 87 storms the solver resolves, and the lowest voltages it predicts overlap (median 0.886 against 0.924 p.u.).

Figure 2. (a) Along a load sweep the AC voltage collapses at the fold; GENCO's prediction barely moves and continues beyond it. (b) At the actual load, GENCO's lowest predicted voltage on storms the solver cannot resolve overlaps with that on storms it can.

3. Why: GENCO under-reacts to system-wide stress

To understand Figure 2a we probed GENCO with small, controlled changes and compared its response with the AC response. Locally, it behaves like the physics: changing a single load moves the voltage at that bus and its neighbours by the right amount (median ratio 0.98 to 1.02). Globally, it does not: when every load grows together, its voltage drops are only 0.60 of the AC drops (median over 129 scenarios). A model that damps system-wide stress will glide past a voltage-collapse point instead of signalling it, which is exactly what Figure 2a shows. Because the flaw is this specific, it can be trained away: Section 6 does so.

Figure 3. (a) GENCO's worst bus-voltage error on random and clustered outages. (b) Voltage drop per 1 % uniform load growth (linearised from ±2 % changes), GENCO against AC, on the same 129 held-out scenarios for the released model and for the model fine-tuned with a load-growth term (Section 6).

4. The fix: GENCO proposes, the solver decides

Our verification module, genco_verify (Figure 1), uses GENCO's state as the first start of a Newton-Raphson solve (lightsim2grid, KLU) to a mismatch below 1e-8 p.u., and falls back to up to three classical starts if it fails. Decisions are taken only on a converged AC state. If no start converges, the case is unresolved: an alarm for an engineer, never a pass.

To test this honestly, we froze the policy, every threshold and 174 operating points that no earlier experiment had touched, then built 870 N-10 and N-20 contingencies on them, random and clustered.

* The other 2 cases have two AC solutions; GENCO's start reached the high-voltage one. A static power flow cannot say which the grid would settle on, so the module reports such cases instead of choosing silently.

Figure 4. (a) Outcome of each of the 870 frozen cases, and the number GENCO alone gets wrong. (b) Median end-to-end time per case.

GENCO's misses concentrate where they hurt. On clustered outages it misses 42 % of the overloads, even with a calibrated margin, and 91 % of the voltage violations. On random N-2 outages it is a good screen: with a conformal margin of 4.54 % of rating it catches every overload in 98.9 % of overloaded scenarios (95 % CI 97.1 to 99.7 %), with 2.6 false alarms per scenario against 7.3 for DC power flow at the same guarantee. It does not save time on this grid: its start saves about one Newton iteration, but inference costs more than the classical solve (13 ms per case). Its value is a better first guess and, as Section 6 shows, a warning.

We also checked the solver itself. PYPOWER, built from the raw network parameters rather than our admittance matrix, reproduces all 46 released solutions it was given and all 21 continuation states near the folds, and converges for none of 99 unresolved N-2 cases.

5. A second grid: IEEE 118

To check that this is not a property of one grid, we repeated the frozen test and the load-growth probe on IEEE 118 with the model the authors released for it (gridfm/genco-pf-datakit), on 118 operating points from their own test split, 708 contingencies in all. Before building any contingency we confirmed that GENCO reproduces the authors' test accuracy (voltage RMSE 0.00031 against their 0.00028 p.u.).

The safety result carries over unchanged. GENCO's weaknesses carry over too, more mildly on the smaller grid, and on both grids its answer gives no sign of the cases the solver cannot resolve.

6. Extending GENCO: fixing the flaw, and reading a warning

Fine-tuning for clustered outages. We fine-tuned the released checkpoint with the authors' training objective in gridfm-graphkit on 6,737 solver-labelled contingencies (69 minutes on one T4), holding one outage construction out entirely, and repeated the training with a second seed. A second frozen protocol compared both models on paired cases neither had seen.

• Missed decisions per clustered case fall from 6.13 to 4.79 (−22 %, paired 95 % CI of the difference −1.72 to −0.97; −20 % with the second seed), and from 2.01 to 1.56 on the held-out construction.
• The fine-tuned model is a specialist: on ordinary scenarios it is less accurate (median error 0.0013 → 0.0036 p.u.). Because whether an outage is clustered is known before any model runs, a rule fixed on the training cases (at least 5 lines out, spread over at most 11 hops) sends clustered outages to the fine-tuned model and everything else to the released one. It routes 400 of 401 clustered frozen cases and none of the 347 random ones, keeping the full gain where it matters and the released accuracy elsewhere.
• Replaying ordinary scenarios during training, under a rule fixed in advance, did not remove this price: it gave up most of the clustered gain, and none of its 20 checkpoints stayed within 10 % of the released model's ordinary accuracy. Routing is how to get both.

Fixing the flaw found in Section 3. Rather than only adding data, we supervised the response GENCO gets wrong. Every training case is paired with the same case at 2 % more load, and the difference between GENCO's and the AC voltage change is penalised alongside the original objective; the decision rule was fixed before the run. On the 129 held-out scenarios of Section 3, GENCO's response to load growth goes from 0.60 to 0.99 of the AC response (1.01 with a second seed), against 0.78 for the fine-tuning above (Figure 3b). Fixing the physics also improves the decisions: missed decisions per clustered case fall from 6.13 to 4.45 (−27 %, paired 95 % CI of the difference −2.23 to −1.23; −31 % with the second seed), against −22 % for the fine-tuning above, and from 2.01 to 1.35 on the held-out construction. Its price on ordinary scenarios is higher (median error 0.0052 p.u.), and the same routing removes it: random outages stay at 0.14 missed decisions per case.

A warning from inside GENCO. A logistic regression on GENCO's outputs and embeddings predicts which cases the solver will leave unresolved, with AUROC 0.96 on the frozen test (0.94 on the held-out construction and on the storms), catching 95 % of them at 16 % false alarms. The outage topology alone, with boosted trees, reaches 0.90 at 35 % false alarms; GENCO adds +0.06 (paired 95 % CI +0.03 to +0.09).

The signal sits in GENCO's internal embeddings, not in its predicted voltages. Embeddings alone beat GENCO's own outputs by +0.07 AUROC on the frozen test (paired 95 % CI +0.03 to +0.10) and by +0.15 on the storms, the very cases where its voltages overlap with those of storms the solver resolves. GENCO knows more than it shows, and that knowledge can be read out to tell an operator which cases to look at first. With the 870 frozen cases ordered by the warning, 95 % of those the solver cannot resolve are in the first 24 % of the queue; ordered by GENCO's power mismatch they take 53 %, and by its lowest predicted voltage 92 %. This matters when an N-k set is too large to check every case in time.

Figure 5. (a) Decisions missed per case by GENCO alone. (b) Error on the released test scenarios. (c) The unresolved-case warning on the 870 frozen cases. Bars: the published run; filled dots: a second seed; open dots: a first run made before a fix to gradient accumulation.

The solver remains the judge. Within 5 % of a fold the warning is near chance (AUROC 0.57). Trained on points near the fold, with whole storms held out, it reaches only 0.63 (95 % CI 0.60 to 0.67), short of the 0.70 we set in advance: close to collapse, GENCO's internal state carries little sign of it. The load-growth model does not change that: over the last 5 % of load before the fold it follows 14 % of the AC voltage drop on the storms the solver cannot resolve (5 % for the released model), so it still glides past the collapse, only less. Newton never converges beyond a fold from either fine-tuned state (0 of 1,689 load points), so the warning orders the work; it does not replace the check.

7. What this gives an operator

A screen built this way never reports a case as safe unless an AC solution says so, and never hides a case it cannot solve. GENCO supplies the first guess, a fine-tuned specialist that responds to load growth like the physics takes the clustered outages, and the embedding warning ranks the cases an engineer should see first. The same audit (clustered outages, a frozen test, a load-growth probe) can be run on other grid foundation models before they are trusted with screening.

Open science and reproducibility

• GitHub: the genco_verify module with tests, and the Colab notebooks that produced every result, including the replay run that was not adopted and the load-growth runs, each judged by a rule fixed before it ran. They run from the public data and checkpoints, with gridfm-graphkit pinned to 95b90ee and gridfm/genco-pf-contingency to 5c625b0. Every result file is included. scripts/headline_numbers.py recomputes every number in this post from those files, and CI checks it on every commit.
• Hugging Face model: the fine-tuned weights and those of the load-growth model (loadgrowth/), each for both training seeds, the warning heads and the feature matrices behind them.
• Case explorer: for each of the 870 frozen-test cases, GENCO's prediction, the warning, the Newton check and the outcome, on a map of the Texas grid.

Scope. The check is a steady-state AC power flow; dynamics, protection and remedial actions are outside it. Unresolved means that none of the tested starts converged, not a proof that no AC solution exists. Next steps: a response term that also covers the steep part of the collapse near the fold, and the same audit on more grids.