Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
a6c0af5
·
1 Parent(s): 4f5f060

Phase 55: temporal-purge CV is THE SCORE and sole validator (user directive); cv.py consolidated to the purge harness only

Browse files

Protocol (session 59 implementation, user-specified): per scenario, one model
on ALL buildings' pairs with label windows fully outside the scenario window,
full-fleet plan, SCORE = ALL-48 mean. Champion 1972.0 (panel 1839.6).
all48/static/screen/lobo/ablate removed from cv.py - recover at 2000281.
Purge-vs-LOBO paired diagnosis recorded: same-device-fate channel ~-18/origin
on the panel for the linear champion, warm-up harsher via calendar starvation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. EXPERIMENTS.md +43 -0
EXPERIMENTS.md CHANGED
@@ -2897,3 +2897,46 @@ foreign training data helps, self-building identity flatters by ~200.
2897
  Everything is judged against 1877.3 paired per scenario from here, sched-flat
2898
  rule in force. Public best remains 3c7b883 at 1661.45; private selection
2899
  recommendation unchanged (score the hardened head publicly, then select).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2897
  Everything is judged against 1877.3 paired per scenario from here, sched-flat
2898
  rule in force. Public best remains 3c7b883 at 1661.45; private selection
2899
  recommendation unchanged (score the hardened head publicly, then select).
2900
+
2901
+ ## Phase 55 - THE SCORE final: temporal-purge CV, sole validator (user directive)
2902
+
2903
+ The CV settles for good, three user decisions in one afternoon session arc:
2904
+
2905
+ 1. **Protocol** (implemented by session 59): for each scenario, ONE model
2906
+ fitted on ALL buildings' pairs whose 42d label windows lie fully outside
2907
+ the scenario window (`origin+42 <= T or origin >= T+42` - label-overlap
2908
+ purge), whole-fleet prediction, ONE full-fleet plan, cost per scenario.
2909
+ SCORE = ALL-48 mean, floor off. Rationale (user): admits every forecasting
2910
+ method (no fold plumbing), one-sentence protocol, no leak within the
2911
+ scenario, and the temporal shape matches deployment (training data spans
2912
+ before and after most scored origins - phase 43a measured 93.9% window
2913
+ coverage). The building axis is deliberately not validated (user judgment:
2914
+ same battery type across buildings).
2915
+ 2. **Sole validator** (user, verbatim intent): "we only use purge score going
2916
+ forward, nothing else to validate a new model or optimization method."
2917
+ Documented as standing rule. LOBO's chimera-plan critique (24 stitched
2918
+ calibrations, no per-building cost attribution, seams inflate the transfer
2919
+ penalty) was the user's decisive argument against building-holdout cost.
2920
+ 3. **Consolidation**: cv.py now contains ONLY the purge harness +
2921
+ oofcand/oofreport. all48/static/screen/lobo/ablate/wf/self removed;
2922
+ recover from git 2000281. exp_x29/exp_x30 no longer run as-is.
2923
+
2924
+ **Champion on THE SCORE: 1972.0** (warmup 3560.3 / panel 1839.6 / nearend
2925
+ 604.8). Level chain, never cross-compare: self-fit 1661.7 (floor on) <
2926
+ exact-LOBO 1877.3 < 5-fold-OOF 1913.3 < purge 1972.0.
2927
+
2928
+ Paired purge-vs-LOBO diagnosis (59's run; LOBO reproduced 1877.3 exactly
2929
+ before removal): the purge is HARSHER overall (+94.7) - entirely warm-up
2930
+ (3560.3 vs 2875.3: purging the calendar-local +/-42d window starves early
2931
+ scenarios more than building-holdout does) - and modestly softer on the panel
2932
+ (-18.1/origin; sched 8.9 vs 9.2 flat, recall 0.468 vs 0.461, rec12 0.576 vs
2933
+ 0.509). Reading: the same-device-fate channel exists but is ~-18/origin for
2934
+ the LINEAR champion - an order of magnitude below the +134/+216 self-fit
2935
+ flattery - because the label-overlap purge removes the direct channel and
2936
+ logbag cannot memorize identity. The known, accepted residual risk: flexible
2937
+ challengers (rf/gbm/dynamics) are flattered by same-battery training; the
2938
+ receipts are phases 51-54 and they are not to be re-argued unprompted.
2939
+
2940
+ The leaderboard closures (appetite, cert-first, pooling) rest on public
2941
+ measurements and survive any CV change. Public best 3c7b883 at 1661.45;
2942
+ private selection recommendation unchanged.