Commit ·
9d7dc15
1
Parent(s): c3a24db
fix: resolve high-fi submit blocker by switching to from_boundary_resolution preset
Browse filesThe high_fidelity VMEC preset forces minimum 10 modes, which doesn't
converge on our mpol=3/ntor=3 boundaries. Switched submit evaluation
to the from_boundary_resolution preset, which adapts resolution to
the boundary's actual Fourier content and converges reliably (~4s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- docs/P1_ENV_CONTRACT_V1.md +8 -2
- docs/P1_HIGHFI_SUBMIT_BLOCKER.md +243 -0
- server/physics.py +2 -1
docs/P1_ENV_CONTRACT_V1.md
CHANGED
|
@@ -163,11 +163,17 @@ The verifier should stay boundary-based:
|
|
| 163 |
|
| 164 |
Do not treat parameterization-specific logic as verifier truth.
|
| 165 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
Training and evaluation rule:
|
| 167 |
|
| 168 |
- use low-fidelity `run` as the RL inner-loop surface
|
| 169 |
-
- keep
|
| 170 |
-
- do not move
|
| 171 |
|
| 172 |
## 9. Reward V0
|
| 173 |
|
|
|
|
| 163 |
|
| 164 |
Do not treat parameterization-specific logic as verifier truth.
|
| 165 |
|
| 166 |
+
VMEC preset mapping:
|
| 167 |
+
|
| 168 |
+
- `run` steps use the `low_fidelity` VMEC preset (~0.6s, tolerant convergence)
|
| 169 |
+
- `submit` uses the `from_boundary_resolution` VMEC preset (~4s, adaptive convergence matching boundary Fourier resolution)
|
| 170 |
+
- the `high_fidelity` VMEC preset (minimum 10 modes, strict convergence) is not used because it does not converge on the current `mpol=3, ntor=3` boundaries
|
| 171 |
+
|
| 172 |
Training and evaluation rule:
|
| 173 |
|
| 174 |
- use low-fidelity `run` as the RL inner-loop surface
|
| 175 |
+
- keep higher-fidelity `submit` for terminal truth, paired fixture checks, submit-side manual traces, and sparse checkpoint evaluation
|
| 176 |
+
- do not move VMEC-backed submit evaluation into every training step unless the contract is deliberately redefined
|
| 177 |
|
| 178 |
## 9. Reward V0
|
| 179 |
|
docs/P1_HIGHFI_SUBMIT_BLOCKER.md
ADDED
|
@@ -0,0 +1,243 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# P1 High-Fidelity Submit Blocker
|
| 2 |
+
|
| 3 |
+
**Date:** 2026-03-07
|
| 4 |
+
**Status:** RESOLVED. Fix: switched submit preset from `high_fidelity` to `from_boundary_resolution`.
|
| 5 |
+
**Predecessor:** [P1_PARAMETERIZATION_DEEPDIVE.md](P1_PARAMETERIZATION_DEEPDIVE.md)
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## 1. The Problem
|
| 10 |
+
|
| 11 |
+
After repairing the parameterization (4-knob family with `triangularity_scale`),
|
| 12 |
+
low-fidelity `run` steps work correctly but **every high-fidelity `submit` crashes**.
|
| 13 |
+
|
| 14 |
+
The environment is functional (no Python exceptions, graceful degradation via
|
| 15 |
+
`evaluation_failed=True`) but no agent can ever produce a nonzero score because
|
| 16 |
+
the submit path always fails with `"VMEC++ did not converge"`.
|
| 17 |
+
|
| 18 |
+
## 2. Evidence
|
| 19 |
+
|
| 20 |
+
### Smoke test: all 3 reset seeds, immediate submit
|
| 21 |
+
|
| 22 |
+
```
|
| 23 |
+
Seed 0: low-fi feas=0.0507 tri=-0.4747 iota=0.2906
|
| 24 |
+
submit: FAILED fidelity=high reason="VMEC++ did not converge"
|
| 25 |
+
|
| 26 |
+
Seed 1: low-fi feas=0.0507 tri=-0.4747 iota=0.2896
|
| 27 |
+
submit: FAILED fidelity=high reason="VMEC++ did not converge"
|
| 28 |
+
|
| 29 |
+
Seed 2: low-fi feas=0.0507 tri=-0.4747 iota=0.3165
|
| 30 |
+
submit: FAILED fidelity=high reason="VMEC++ did not converge"
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
### Smoke test: heuristic-style episode (seed 0)
|
| 34 |
+
|
| 35 |
+
```
|
| 36 |
+
Start: feas=0.0507 tri=-0.4747 iota=0.2906 failed=False
|
| 37 |
+
tri+s: feas=0.0517 tri=-0.4852 iota=0.2845 r=-0.11 failed=False
|
| 38 |
+
rt+s: feas=0.0296 tri=-0.4852 iota=0.2977 r=+0.01 failed=False
|
| 39 |
+
submit: feas=1000000 score=0.0000 r=-3.00 failed=True
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Low-fi steps produce real physics feedback (feasibility moves, constraint values
|
| 43 |
+
respond to knob changes). But the boundary that passes low-fi evaluation fails
|
| 44 |
+
high-fi evaluation universally.
|
| 45 |
+
|
| 46 |
+
### What the failure handling does correctly
|
| 47 |
+
|
| 48 |
+
The VMEC crash handling from `d585eb2` works as designed:
|
| 49 |
+
|
| 50 |
+
- `evaluate_boundary` catches `RuntimeError` and returns `_failure_metrics`
|
| 51 |
+
- `_failure_metrics` sets `evaluation_failed=True`, `p1_feasibility=1_000_000.0`
|
| 52 |
+
- The environment applies `FAILURE_PENALTY = -2.0` plus terminal penalties
|
| 53 |
+
- `_update_best` skips failed evaluations (doesn't corrupt best-known state)
|
| 54 |
+
- `_reference_metrics` falls back to last successful evaluation for reward deltas
|
| 55 |
+
- The observation text shows `evaluation_status=FAILED` with the failure reason
|
| 56 |
+
|
| 57 |
+
No silent swallowing, no crashes, no state corruption. The problem is purely that
|
| 58 |
+
the physics solver rejects these boundaries at high fidelity.
|
| 59 |
+
|
| 60 |
+
## 3. Why It Happens
|
| 61 |
+
|
| 62 |
+
### Low-fi vs high-fi VMEC settings
|
| 63 |
+
|
| 64 |
+
```python
|
| 65 |
+
low: ConstellarationSettings(vmec_preset_settings=fidelity='low_fidelity')
|
| 66 |
+
high: ConstellarationSettings(vmec_preset_settings=fidelity='high_fidelity')
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
High-fidelity VMEC uses more radial grid points and stricter convergence criteria.
|
| 70 |
+
The 4-knob boundaries (generated at `mpol=3, ntor=3` with injected triangularity)
|
| 71 |
+
are not spectrally smooth enough for the high-fi solver to converge.
|
| 72 |
+
|
| 73 |
+
### Why low-fi works
|
| 74 |
+
|
| 75 |
+
Low-fi VMEC is more tolerant of rough boundaries. It uses fewer grid points and
|
| 76 |
+
relaxed convergence thresholds, so the same boundary that fails high-fi can still
|
| 77 |
+
produce valid physics at low-fi.
|
| 78 |
+
|
| 79 |
+
### The original session handled this differently
|
| 80 |
+
|
| 81 |
+
From `P1_SCORE_CHASE_NOTES.md`, the winning approach used a multi-fidelity pipeline:
|
| 82 |
+
|
| 83 |
+
1. Low-fi VMEC inside the optimization loop (fast, robust)
|
| 84 |
+
2. Periodic promotion to `from_boundary_resolution` (mid gate)
|
| 85 |
+
3. Final promotion to `high_fidelity` (final truth)
|
| 86 |
+
|
| 87 |
+
The boundaries that reached high-fi were already refined through optimization
|
| 88 |
+
(trust-region, ALM+NGOpt) with much higher Fourier resolution (`mpol=8, ntor=8`).
|
| 89 |
+
Our 4-knob `mpol=3, ntor=3` boundaries are too coarse for high-fi convergence.
|
| 90 |
+
|
| 91 |
+
## 4. Fix Options
|
| 92 |
+
|
| 93 |
+
### Option A: Use mid-fidelity for submit
|
| 94 |
+
|
| 95 |
+
Replace the `high_fidelity` preset in `_settings_for_fidelity("high")` with
|
| 96 |
+
`from_boundary_resolution` or another intermediate preset.
|
| 97 |
+
|
| 98 |
+
**Pros:**
|
| 99 |
+
- Minimal code change (one line in `physics.py`)
|
| 100 |
+
- Still more rigorous than low-fi `run` steps
|
| 101 |
+
- Likely to converge on these boundaries
|
| 102 |
+
|
| 103 |
+
**Cons:**
|
| 104 |
+
- Submit results are no longer comparable to the official leaderboard evaluator
|
| 105 |
+
- Need to verify `from_boundary_resolution` actually exists and converges
|
| 106 |
+
|
| 107 |
+
**Verdict:** Pragmatic for the hackathon. The environment already documents that
|
| 108 |
+
it's a stepping stone, not a leaderboard tool.
|
| 109 |
+
|
| 110 |
+
### Option B: Increase Fourier resolution in boundary construction
|
| 111 |
+
|
| 112 |
+
Change `build_boundary_from_params` to use `mpol=5, ntor=5` or higher instead
|
| 113 |
+
of `mpol=3, ntor=3`. More Fourier modes produce a smoother boundary that high-fi
|
| 114 |
+
VMEC can converge on.
|
| 115 |
+
|
| 116 |
+
**Pros:**
|
| 117 |
+
- Addresses root cause (boundary too coarse)
|
| 118 |
+
- Submit uses the real high-fi evaluator
|
| 119 |
+
|
| 120 |
+
**Cons:**
|
| 121 |
+
- Higher modes may change which parameter regions are feasible
|
| 122 |
+
- Need to re-run the 4-knob sweep to verify feasibility still holds
|
| 123 |
+
- Low-fi `run` steps become slower (more modes = more computation)
|
| 124 |
+
|
| 125 |
+
**Verdict:** Correct fix but requires re-validation. May invalidate current
|
| 126 |
+
ranges, deltas, and seed pool.
|
| 127 |
+
|
| 128 |
+
### Option C: Both (higher modes + mid-fi submit)
|
| 129 |
+
|
| 130 |
+
Increase `mpol/ntor` moderately (e.g., 4 or 5) AND use a mid-fi submit preset.
|
| 131 |
+
|
| 132 |
+
**Pros:**
|
| 133 |
+
- Belt and suspenders: smoother boundary + more tolerant solver
|
| 134 |
+
- Most likely to unblock submit without breaking low-fi behavior
|
| 135 |
+
|
| 136 |
+
**Cons:**
|
| 137 |
+
- Two changes at once make it harder to attribute any regressions
|
| 138 |
+
- Need re-validation of the parameter landscape
|
| 139 |
+
|
| 140 |
+
### Option D: Use same fidelity for both run and submit
|
| 141 |
+
|
| 142 |
+
Make both `run` and `submit` use the same low-fi settings. Submit costs a budget
|
| 143 |
+
step but doesn't add fidelity uplift.
|
| 144 |
+
|
| 145 |
+
**Pros:**
|
| 146 |
+
- Guaranteed to work (low-fi already converges)
|
| 147 |
+
- Simplest possible fix
|
| 148 |
+
|
| 149 |
+
**Cons:**
|
| 150 |
+
- Loses the multi-fidelity story entirely
|
| 151 |
+
- Submit becomes meaningless as a distinct action (same eval as run)
|
| 152 |
+
- The low-fi/high-fi split was a deliberate design choice
|
| 153 |
+
|
| 154 |
+
**Verdict:** Only if nothing else works. Last resort.
|
| 155 |
+
|
| 156 |
+
## 5. Resolution
|
| 157 |
+
|
| 158 |
+
**Option A was applied.** The `from_boundary_resolution` preset exists in
|
| 159 |
+
`constellaration.mhd.vmec_settings` and is actually the library's default preset.
|
| 160 |
+
|
| 161 |
+
### Investigation results
|
| 162 |
+
|
| 163 |
+
1. `from_boundary_resolution` adapts VMEC resolution to the boundary's Fourier
|
| 164 |
+
resolution and "optimizes for convergence rate over runtime and high fidelity"
|
| 165 |
+
2. `high_fidelity` forces minimum 10 poloidal and toroidal modes regardless of
|
| 166 |
+
boundary resolution — far beyond our `mpol=3, ntor=3` boundaries
|
| 167 |
+
3. All 3 reset seeds converge with `from_boundary_resolution` (~4s each)
|
| 168 |
+
4. Tested across 6 diverse parameter combos: `from_boundary_resolution` converges
|
| 169 |
+
everywhere that `low_fidelity` converges — no case where low-fi works but mid-fi fails
|
| 170 |
+
5. Metrics are nearly identical between low-fi and mid-fi (slight iota differences,
|
| 171 |
+
matching triangularity and feasibility)
|
| 172 |
+
|
| 173 |
+
### Fix
|
| 174 |
+
|
| 175 |
+
One-line change in `server/physics.py`:
|
| 176 |
+
|
| 177 |
+
```python
|
| 178 |
+
# Before (crashed on all 4-knob boundaries):
|
| 179 |
+
vmec_preset_settings=ConstellarationSettings.default_high_fidelity_skip_qi().vmec_preset_settings
|
| 180 |
+
|
| 181 |
+
# After (converges on all boundaries that low-fi converges on):
|
| 182 |
+
vmec_preset_settings=VmecPresetSettings(fidelity="from_boundary_resolution")
|
| 183 |
+
```
|
| 184 |
+
|
| 185 |
+
### Available VMEC presets (for reference)
|
| 186 |
+
|
| 187 |
+
| Preset | Purpose | Convergence | Speed |
|
| 188 |
+
|--------|---------|-------------|-------|
|
| 189 |
+
| `very_low_fidelity` | Fast optimization | Very tolerant | ~0.3s |
|
| 190 |
+
| `low_fidelity` | Runtime over fidelity (our `run` steps) | Tolerant | ~0.6s |
|
| 191 |
+
| `from_boundary_resolution` | Match boundary resolution (our `submit`) | Adaptive | ~4s |
|
| 192 |
+
| `high_fidelity` | Max correctness (min 10 modes) | Strict | crashes on coarse boundaries |
|
| 193 |
+
|
| 194 |
+
### Verification
|
| 195 |
+
|
| 196 |
+
Full environment episode with submit now works:
|
| 197 |
+
|
| 198 |
+
```
|
| 199 |
+
Reset: feas=0.0507 tri=-0.4747 iota=0.2906
|
| 200 |
+
tri+s: feas=0.0517 tri=-0.4852 iota=0.2845 r=-0.105 failed=False
|
| 201 |
+
rt+s: feas=0.0296 tri=-0.4852 iota=0.2977 r=+0.011 failed=False
|
| 202 |
+
tri+s: feas=0.0292 tri=-0.4954 iota=0.2912 r=-0.098 failed=False
|
| 203 |
+
submit: feas=0.0322 score=0.0000 r=-1.015 failed=False fidelity=high
|
| 204 |
+
```
|
| 205 |
+
|
| 206 |
+
Submit produces real physics metrics, no crash, correct fidelity labeling.
|
| 207 |
+
|
| 208 |
+
## 6. What This Does NOT Invalidate
|
| 209 |
+
|
| 210 |
+
- The 4-knob parameterization repair is correct (low-fi feasibility works)
|
| 211 |
+
- The VMEC crash handling is correct (graceful degradation, no state corruption)
|
| 212 |
+
- The reward V0 design is correct (failure penalty, reference metrics fallback)
|
| 213 |
+
- The heuristic baseline design is correct (reactive constraint repair strategy)
|
| 214 |
+
- The reset seed pool design is correct (diverse near-feasible starts)
|
| 215 |
+
|
| 216 |
+
The only thing broken is the submit fidelity gate. Everything else in `d585eb2`
|
| 217 |
+
is validated by the smoke tests.
|
| 218 |
+
|
| 219 |
+
## 7. Affected Files
|
| 220 |
+
|
| 221 |
+
The fix will touch:
|
| 222 |
+
|
| 223 |
+
- `server/physics.py` — `_settings_for_fidelity("high")` and/or
|
| 224 |
+
`build_boundary_from_params` defaults
|
| 225 |
+
- Possibly `server/environment.py` if submit semantics change
|
| 226 |
+
- `docs/P1_ENV_CONTRACT_V1.md` — if the multi-fidelity contract changes
|
| 227 |
+
|
| 228 |
+
## 8. Baseline Results (from background runs)
|
| 229 |
+
|
| 230 |
+
The heuristic and random baselines completed their 20-episode runs. Both ran
|
| 231 |
+
on the repaired parameterization with low-fi `run` steps working correctly.
|
| 232 |
+
Submit steps in both baselines triggered the high-fi crash, so all episodes
|
| 233 |
+
ended with `evaluation_failed=True` on submit and zero final scores.
|
| 234 |
+
|
| 235 |
+
The low-fi step behavior was healthy:
|
| 236 |
+
|
| 237 |
+
- Heuristic agent correctly identified and repaired constraint violations
|
| 238 |
+
- Random agent produced varied trajectories with some feasibility improvement
|
| 239 |
+
- VMEC crash handling worked throughout (no Python exceptions, graceful penalties)
|
| 240 |
+
- The `restore_best` action worked correctly after failures
|
| 241 |
+
|
| 242 |
+
The baselines are ready to produce meaningful results once the submit blocker
|
| 243 |
+
is resolved.
|
server/physics.py
CHANGED
|
@@ -9,6 +9,7 @@ from constellaration.forward_model import (
|
|
| 9 |
ConstellarationSettings,
|
| 10 |
forward_model,
|
| 11 |
)
|
|
|
|
| 12 |
from constellaration.geometry import surface_rz_fourier
|
| 13 |
from constellaration.geometry.surface_rz_fourier import SurfaceRZFourier
|
| 14 |
from constellaration.initial_guess import generate_rotating_ellipse
|
|
@@ -94,7 +95,7 @@ def evaluate_boundary(
|
|
| 94 |
def _settings_for_fidelity(fidelity: EvaluationFidelity) -> ConstellarationSettings:
|
| 95 |
if fidelity == "high":
|
| 96 |
return ConstellarationSettings(
|
| 97 |
-
vmec_preset_settings=
|
| 98 |
boozer_preset_settings=None,
|
| 99 |
qi_settings=None,
|
| 100 |
turbulent_settings=None,
|
|
|
|
| 9 |
ConstellarationSettings,
|
| 10 |
forward_model,
|
| 11 |
)
|
| 12 |
+
from constellaration.mhd.vmec_settings import VmecPresetSettings
|
| 13 |
from constellaration.geometry import surface_rz_fourier
|
| 14 |
from constellaration.geometry.surface_rz_fourier import SurfaceRZFourier
|
| 15 |
from constellaration.initial_guess import generate_rotating_ellipse
|
|
|
|
| 95 |
def _settings_for_fidelity(fidelity: EvaluationFidelity) -> ConstellarationSettings:
|
| 96 |
if fidelity == "high":
|
| 97 |
return ConstellarationSettings(
|
| 98 |
+
vmec_preset_settings=VmecPresetSettings(fidelity="from_boundary_resolution"),
|
| 99 |
boozer_preset_settings=None,
|
| 100 |
qi_settings=None,
|
| 101 |
turbulent_settings=None,
|