As Peder encountered issues with his Hugging Face account, I am posting this on his behalf.

GRID MIRAGE — Same Physics. Different Representation. Different Predictions?

Participant: Peder Falkman, CEO, Solid Soup — LinkedIn

Contact: peder@blas.se

Code, results, report (Apache-2.0): https://github.com/PederFalkman/grid-mirage-genco

Summary

We propose GRID MIRAGE, a label-free acceptance test for grid foundation models, and run it on eight released GENCO power-flow checkpoints: IEEE 14 and IEEE 118, Tiny/Small/Base, and two training seeds for IEEE 118. Each checkpoint is evaluated on 50 scenarios from its own official held-out test split. The test asks three questions that point accuracy cannot answer:

• Mirror: if the same grid is numbered differently, does the model give the same answer?
• Butterfly: when demand changes a little, does the model move the grid state as far as the physics does?
• Physics Lens: do its predictions still balance power where the model does not enforce it?

Findings, all measured on real GENCO inference:

1. Mirror passes on all 8 checkpoints. Relabeling changes float64 predictions by at most 2e-14.
2. Butterfly depends on the grid. All IEEE 14 checkpoints respond correctly (gain ≈ 0.99). All IEEE 118 checkpoints under-react: their voltages, angles and line flows move only 71–89 % as far as the physics does.
3. On IEEE 118, larger models under-react more, in both seeds. The voltage-angle gain is 0.80–0.86 for Tiny and 0.71–0.72 for Base. Point accuracy does not predict this: a Tiny checkpoint with Base-level residual still responds far better.
4. The bias is correctable. One Newton–Raphson step from GENCO's prediction restores a gain of 1.00 on every checkpoint.

Figure 1 — Three label-free tests on official held-out scenarios, plus a hybrid check of whether a detected bias is correctable.

Motivation

Two properties matter before an operator or planner relies on a learned power-flow solver, and neither is visible in a test-set error:

1. Representation independence. The same physical grid arrives with different bus numbering from different EMS exports, planning tools and data pipelines. A trustworthy model must give the same physical answer for every valid numbering.
2. Correct sensitivities. Most engineering questions are what-if questions: what happens if load grows 1 %, if a transfer increases, if a line trips? They depend on how much the state changes, not only on where it is. A model can be accurate on average and still systematically under- or over-react.

Both can be tested without new labels: relabeling and small perturbations come with a mathematically known correct answer (equivariance, and the AC power-flow equations).

Method

• Models and data:
  • Checkpoints: the official gridfm/genco-pf-datakit checkpoints: case14_ieee and case118_ieee, each in tiny/small/base (hidden size 12/24/48), seed 0, plus seed 1 for IEEE 118 Tiny and Base.
  • Pipeline: each is loaded with GraphKit's own config, normalizer, PF masking and strict state-dict loading (genco-paper-repro, commit 4e7fbcf), and runs on CPU in eval mode.
  • Data: for each checkpoint, 50 scenarios come from its own official test split of gridfm/pf_small_case14_ieee / gridfm/pf_small_case118_ieee, replicated with GraphKit's own split function, so there is no training leakage.
• Mirror: five random but physically consistent relabelings per case. Bus rows, generator rows, gen↔bus edges and branch order are all permuted together, and outputs are mapped back to the original numbering. Every case runs in float32 and float64, to separate model behaviour from round-off.
• Butterfly: all demand scaled by −1, −0.5, −0.1, 0, +0.1, +0.5, +1 %, with generator set-points fixed and the slack absorbing the change. GENCO's predicted change is compared with an independent AC Newton–Raphson solution of the same input on the scenario's own admittances. That reference reproduces the dataset's AC labels in all 400 scenarios and matches pandapower to 1e-14. The response gain is the projection of GENCO's change onto the true change: 1 means a correct response, below 1 means under-reaction.
• Physics Lens: AC nodal power balance of GENCO's predictions, only on quantities the model does not enforce by construction (P at PQ/PV buses, Q at PQ buses). GENCO's decoder sets slack-bus P and PV-bus Q from the predicted voltages, and including those would flatter the result.
• Hybrid check: one Newton–Raphson step warm-started from GENCO's prediction, re-tested with Butterfly and Physics Lens.

In total: 8 checkpoints × 50 scenarios × 7 load levels × 6 representations × 2 precisions = 33,600 GENCO forward passes, under 2 hours on a 4-core CPU.

Results

Figure 2 — Butterfly response gain for every released checkpoint tested, with the mean, 95 % CI and 5th–95th percentile over 50 held-out scenarios. 1 = exact physics.

Mirror: GENCO sees through the mirage. On every checkpoint, relabeling changes float64 predictions by at most 2e-14. In float32 the spread (up to 1e-5 rad and 0.07 Mvar on IEEE 118) is pure summation-order round-off: identical inputs give bit-identical outputs, and merely reordering the edge list produces the same magnitude. The released models and preprocessing are safe to feed grids with any bus numbering, consistent with the architecture having no positional encodings or bus-ID features.

Butterfly: the response is correct on IEEE 14 and attenuated on IEEE 118.

• IEEE 14: all three model sizes track the physics (gains 0.99–1.00).
• IEEE 118: every checkpoint under-reacts. Its voltage-angle gain is below 1 in all 50 scenarios of every IEEE 118 checkpoint. On IEEE 118 Base, the median line whose true flow moves by more than 0.5 MW moves only 0.82× as far in GENCO's prediction.
• Linear, not an extrapolation effect: the gain is the same from ±0.1 % to ±1 % and symmetric in sign.
• Model size: within IEEE 118, the gain falls with model size in both seeds (Va: Tiny 0.80/0.86 → Base 0.72/0.71, with non-overlapping CIs).
• Not explained by accuracy: Tiny seed 1 reaches about the same power-balance residual as Base (0.11 vs 0.12–0.13 MW), yet responds with gain 0.86 instead of 0.71. Standard accuracy metrics therefore do not reveal this sensitivity bias.

Figure 3 — IEEE 118 Base. Left: predicted vs true voltage-angle change for ±1 % load at every non-slack bus of 50 held-out scenarios. GENCO (orange) lies below the physics line, and one Newton step from GENCO (blue) lies on it. Right: mean power-balance residual.

Physics Lens. Our residual at the dataset operating point is close to each checkpoint's published metric (table, last column). On IEEE 118 Base it grows from 0.125 MW at the dataset point to 0.137–0.145 MW at ±1 % load.

Hybrid check: the bias is correctable. One Newton–Raphson step from GENCO's prediction brings the mean gain on every checkpoint to 1.00 (every individual case within 0.98–1.02). It cuts the residual about 200× (e.g. 0.13 → 0.0006 MW on IEEE 118 Base). As an initializer, GENCO roughly halves the iterations of a full solve (4.0 → 1.9–2.2). We do not claim a wall-clock speed-up.

Per-checkpoint Mirror and Butterfly figures are in the repository (results/genco_*), for example IEEE 118 Base Mirror and IEEE 14 Tiny Butterfly.

How this connects to other entries

• GENCO-PTDF (#2) finds GENCO's AC-PTDF sensitivities off by about 10 % on IEEE 118 and improves them with response-supervised fine-tuning. Butterfly shows the direction of that error under a system-wide load change (attenuation), shows that it is grid- and size-dependent, and needs neither labels nor training. It can serve as a quick check of whether such fine-tuning has worked.
• Anchored screening (#8) observes that GENCO's errors are largely systematic, and starts Newton from GENCO's predicted change. Our measurement quantifies part of that systematic error: on IEEE 118, GENCO's predicted change is itself 11–29 % too small. That is consistent with their finding that a Newton step is still needed after anchoring.
• The GENCO paper envisions a safeguarded hybrid workflow with AC re-evaluation (section 6). Butterfly provides a measurable acceptance criterion for when that safeguard is required: on IEEE 14 the released models pass, while on IEEE 118 they need it.

Conclusion

Representation is not GENCO's weakness: every released checkpoint we tested is exactly equivariant to bus relabeling. Physical sensitivity is where they differ. GENCO tracks the physics on IEEE 14 but under-reacts by 11–29 % on IEEE 118, more so for the larger models, and point-accuracy metrics hide this. A label-free metamorphic test exposes it in minutes per checkpoint, and one Newton step removes it. We propose Butterfly and Mirror as standard acceptance tests for grid foundation models, and GENCO plus one physics step as the safe default wherever sensitivities matter on larger grids.

Next steps

1. Contribute Mirror and Butterfly as reusable checks to GraphKit, so every released checkpoint reports its equivariance error and response gain next to its accuracy.
2. Extend to larger grids (GOC 500/2000) and to other perturbation directions (single-bus injections, transfers, contingencies) to map where the attenuation starts.
3. Investigate the cause (e.g. a smoothing effect of the regression objective growing with model capacity) and use the response gain as a training signal, as in #2's response supervision.

Reproducibility

Everything is public and pinned:

• Checkpoints: gridfm/genco-pf-datakit, with SHA-256 hashes in each results/*/metadata.json.
• Datasets: pf_small_case14_ieee at revision 5669b8ce, pf_small_case118_ieee at revision 3f5a6e26.
• GraphKit: commit 4e7fbcf.
• Splits: the official test splits; every scenario ID is recorded.

python src/fetch_genco_inputs.py downloads only the ~1.3 GB of inputs needed, and bash run_all_checkpoints.sh reproduces every number and figure. The repository contains the code, unit tests, raw CSVs for every case, bootstrap summaries, figures and pip freeze: https://github.com/PederFalkman/grid-mirage-genco (Apache-2.0).

All results above are real GENCO inference on released checkpoints. The repository also contains a reference-only check of our AC solver on the standard IEEE 118 case and a six-bus DC toy example; these are labelled as such and are not GENCO results.