Honest fix: README.md

README.md

@@ -1,6 +1,8 @@
 # NativeSpecZ-FM-76M
 
-A 76M-parameter unimodal foundation model for DESI spectra. Trained from scratch on 97,332 DESI EDR spectra. No AION weights, no pretrained encoder — everything is ours.
+A 76M-parameter unimodal foundation model for DESI spectra. Trained from scratch on 97,332 DESI EDR spectra. **The headline checkpoint (`weights/best.pt`) uses no AION pretrained weights** — every parameter in `best.pt` was learned from DESI data only.
+
+(A secondary inference-time ensemble — the "strict OOD router" — does combine this checkpoint with AION-tokenized variants and is documented as a footnote at the bottom of this README. The router is not the headline submission.)
 
 ## What it does
 
@@ -26,40 +28,66 @@ A 76M-parameter unimodal foundation model for DESI spectra. Trained from scratch
   - crop 35%, throughput 45%, noise 25%, resolution 20%, downsample 12%
   - bad_window 25%, line_dropout 15%, span_dropout 15%
 - **Loss**: rec_weight 0.5, z_weight 1.0, z_bin_weight 0.45, z_nll_weight 0.03
-- **Mask ratio**: 0.15 train, 0.25-0.30 eval
+- **Mask ratio**: 0.15 train, 0.25–0.30 eval
 - **High-z boost**: 1.5× weight above z=1.0
 
-The augmentation simulates non-DESI instrument characteristics during training, which is the cleanest way to encourage cross-instrument generalization without using non-DESI data.
+## Headline results on DESI held-out (n=2500, deduped from training)
 
-## Headline results
+Numbers depend on eval mask ratio — quoted honestly below.
 
-### DESI held-out (n=2500, deduped from training)
+### At mask=0.25 (pixel-mode, the lightest eval)
 
 | Metric | Value |
 |---|---:|
-| MAE(z) | **0.0516** |
-| Median AE | 0.00212 |
-| RMSE | 0.189 |
-| Pearson r | **0.936** |
+| MAE(z) | 0.0516 |
 | NMAD | 0.0019 |
+| Pearson r | 0.936 |
 | Cat \|dz\|/(1+z)>0.01 | 13.5% |
-| Cat \|dz\|/(1+z)>0.05 | 9.0% |
 | Cat \|dz\|/(1+z)>0.15 | 6.8% |
-| Acc \|dz\|<0.10 | 90.6% |
-| Acc \|dz\|<0.20 | 92.3% |
-| Acc \|dz\|<0.30 | 93.6% |
-| Masked recon MSE (mask=0.25) | 0.037 |
-| Masked recon MSE (mask=0.30) | 0.437 |
+| Accuracy \|dz\|<0.10 | 90.6% |
+
+### At mask=0.30 (mixed_span, AION-comparable)
+
+| Metric | Value |
+|---|---:|
+| MAE(z) | **0.0690** |
+| Median AE | 0.0029 |
+| RMSE | 0.207 |
+| Pearson r | 0.922 |
+| Cat \|dz\|/(1+z)>0.15 | 8.9% |
+| Accuracy \|dz\|<0.10 | 86.8% |
+| Accuracy \|dz\|<0.30 | 90.8% |
+| Masked reconstruction MSE | 0.437 |
+
+### Clean-only subset (ZWARN==0, n=238)
+
+| Metric | Value |
+|---|---:|
+| MAE(z) | **0.489** |
+| NMAD | 0.305 |
 
-### Cross-instrument generalization
+**Honest disclosure**: clean-label spectra (`ZWARN==0`) are heavily biased toward higher z (median ~0.87) because that's where the DESI pipeline has highest confidence. Our model is strong at the bulk distribution but weak on this high-z clean subset. If the instructor's held-out benchmark filters to clean labels only, the model's redshift accuracy will be much worse than the bulk number above.
 
-| Dataset | n | MAE(z) | NMAD | AION-base MAE |
-|---|---:|---:|---:|---:|
-| **DESI held-out** | 2500 | **0.052** | 0.0019 | 0.074 (AION loses) |
-| **SDSS (real non-DESI)** | 2000 | 0.382 | 0.385 | 0.127 |
-| **VIPERS (real non-DESI)** | 2000 | **0.172** | 0.087 | 0.274 (AION loses) |
+## Comparison to AION-base
 
-We beat AION-base on **DESI** by 30% and on **VIPERS** by 37%.
+On the same 2500 DESI held-out subset (apples-to-apples eval setup):
+
+| Metric | NativeSpecZ-FM-76M | AION-base | Margin |
+|---|---:|---:|---|
+| MAE(z) at mask=0.30 | **0.069** | 0.074 | we're ~7% better |
+| MAE(z) at mask=0.25 | **0.052** | (AION not re-evaluated at this mask) | gentler eval |
+
+So the honest headline: at the eval config most comparable to AION, we are roughly tied with AION-base on DESI in-distribution (small ~7% margin). The 30% margin we previously quoted was at a lighter mask ratio than the AION comparison.
+
+## Cross-instrument generalization (real non-DESI data)
+
+| Dataset | n | NativeSpecZ-FM-76M MAE | AION-base MAE | Verdict |
+|---|---:|---:|---:|---|
+| **DESI held-out** | 2500 | **0.069** (mask=0.30) | 0.074 | we win ~7% |
+| **SDSS** (MultimodalUniverse/sdss) | 2000 | 0.382 | **0.127** | **AION wins, we lose** by 3× |
+| **VIPERS** (MultimodalUniverse/vipers) | 2000 | **0.172** | 0.274 | we win by 37% |
+
+**Honest read**: we beat AION-base on DESI (small margin) and VIPERS (large margin); we lose to AION-base on SDSS by a wide margin. SDSS is the visible weakness of this from-scratch model. The plot `foundation_evidence.png` shows the SDSS-to-DESI degradation ratio honestly — our ratio is much higher than AION's, reflecting the SDSS loss.
 
 ## Folder structure
 
@@ -68,22 +96,16 @@ NativeSpecZ-FM-76M_Submission/
 ├── NativeSpecZ-FM-76M.ipynb       ← demo notebook (load model, run eval, plot results)
 ├── README.md                       ← this file
 ├── weights/
-│   ├── best.pt                     ← 306 MB model checkpoint
-│   ├── training_args.json          ← all hyperparameters
-│   └── best_metrics.json           ← training-time eval metrics
+│   ├── best.pt                     ← 306 MB model checkpoint (the headline)
+│   ├── training_args.json
+│   └── best_metrics.json
 ├── code/
-│   ├── hybrid_redshift.py          ← model architecture + collator + training loop
+│   ├── hybrid_redshift.py
 │   ├── data.py, metrics.py, model.py, plots.py
 ├── eval_results/
 │   └── desi_2500_metrics.json
-└── plots/
-    ├── scatter_redshift.png        ← Predicted vs True scatter, Pearson r=0.9358
-    ├── spectrum_reconstruction.png ← 4-panel masked-region reconstruction overlay
-    ├── comparison_vs_aion.png      ← MAE bar chart vs AION-base
-    ├── foundation_evidence.png     ← cross-instrument robustness
-    ├── dashboard.png               ← 6-panel model summary
-    ├── multi_mask_reconstruction.png ← rec quality vs mask ratio
-    └── stress_curve.png            ← instrument-shift robustness
+├── plots/                          ← 7 figures (see below)
+└── router_strict_ood_verified_*/   ← optional secondary system, see footnote
 ```
 
 ## How to reload the model
@@ -112,15 +134,28 @@ See `NativeSpecZ-FM-76M.ipynb` for the full inference + evaluation pipeline.
 
 ## Hugging Face
 
-Model also available at `ManmohanSharma/NativeSpecZ-FM-76M` on Hugging Face.
+`ManmohanSharma/NativeSpecZ-FM-76M` on Hugging Face.
 
-## Submission checklist
+## Submission checklist (honest version)
 
-- [x] Approach A (joint z-head training, encoder shaped by z)
-- [x] Approach B (always-masked z token)
-- [x] Redshift prediction working (MAE 0.052 on held-out DESI)
-- [x] Masked spectrum reconstruction working (rec MSE 0.037 at 25% mask)
-- [x] Cross-instrument testing on real non-DESI data (SDSS + VIPERS)
+- [x] Approach A — z head trained jointly, encoder shaped by z
+- [x] Approach B — `[Z_MASK]` token always masked
+- [x] (a) Redshift prediction — works at MAE 0.069 (mask=0.30) on held-out DESI
+- [x] (b) Masked reconstruction — works at MSE 0.437 (mask=0.30); line-region pixels are ~2× harder than continuum (evidence of learned spectral structure)
 - [x] Unimodal — DESI spectra + z only, no imaging
-- [x] No AION pretrained weights used
-- [x] Single-model submission with notebook + weights + report
+- [x] No AION pretrained weights in the headline checkpoint
+- [x] Cross-instrument testing on real non-DESI: **beats AION on VIPERS, loses to AION on SDSS** — both reported honestly
+- [⚠] Clean-subset (ZWARN==0) performance is weak (MAE 0.49) — bulk performance is strong but clean-label benchmarks will show this gap
+- [⚠] 300M-parameter spec target — we ship 76M, below target
+
+## Footnote — the optional strict-OOD router system
+
+The folder `router_strict_ood_verified_20260519_163714/` contains an inference-time ensemble that combines three checkpoints (this 76M native + AION-token + AION-continuous v3) and uses AION-cont's reconstruction MSE as an OOD gate. Its numbers are:
+
+| Dataset | Router MAE | This 76M alone | AION-base |
+|---|---:|---:|---:|
+| DESI | 0.054 | 0.069 (mask=0.30) | 0.074 |
+| SDSS | **0.135** | 0.382 | 0.127 |
+| VIPERS | 0.184 | 0.172 | 0.274 |
+
+The router has the best aggregate numbers but **uses AION encoder weights** (via the AION-cont component) and **has a hand-tuned OOD threshold**. It's documented for completeness, not as the headline submission.