Thibaut Forest Claude Opus 5 (1M context) commited on
Commit ·
a6c0af5
1
Parent(s): 4f5f060
Phase 55: temporal-purge CV is THE SCORE and sole validator (user directive); cv.py consolidated to the purge harness only
Browse filesProtocol (session 59 implementation, user-specified): per scenario, one model
on ALL buildings' pairs with label windows fully outside the scenario window,
full-fleet plan, SCORE = ALL-48 mean. Champion 1972.0 (panel 1839.6).
all48/static/screen/lobo/ablate removed from cv.py - recover at 2000281.
Purge-vs-LOBO paired diagnosis recorded: same-device-fate channel ~-18/origin
on the panel for the linear champion, warm-up harsher via calendar starvation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- EXPERIMENTS.md +43 -0
EXPERIMENTS.md
CHANGED
|
@@ -2897,3 +2897,46 @@ foreign training data helps, self-building identity flatters by ~200.
|
|
| 2897 |
Everything is judged against 1877.3 paired per scenario from here, sched-flat
|
| 2898 |
rule in force. Public best remains 3c7b883 at 1661.45; private selection
|
| 2899 |
recommendation unchanged (score the hardened head publicly, then select).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2897 |
Everything is judged against 1877.3 paired per scenario from here, sched-flat
|
| 2898 |
rule in force. Public best remains 3c7b883 at 1661.45; private selection
|
| 2899 |
recommendation unchanged (score the hardened head publicly, then select).
|
| 2900 |
+
|
| 2901 |
+
## Phase 55 - THE SCORE final: temporal-purge CV, sole validator (user directive)
|
| 2902 |
+
|
| 2903 |
+
The CV settles for good, three user decisions in one afternoon session arc:
|
| 2904 |
+
|
| 2905 |
+
1. **Protocol** (implemented by session 59): for each scenario, ONE model
|
| 2906 |
+
fitted on ALL buildings' pairs whose 42d label windows lie fully outside
|
| 2907 |
+
the scenario window (`origin+42 <= T or origin >= T+42` - label-overlap
|
| 2908 |
+
purge), whole-fleet prediction, ONE full-fleet plan, cost per scenario.
|
| 2909 |
+
SCORE = ALL-48 mean, floor off. Rationale (user): admits every forecasting
|
| 2910 |
+
method (no fold plumbing), one-sentence protocol, no leak within the
|
| 2911 |
+
scenario, and the temporal shape matches deployment (training data spans
|
| 2912 |
+
before and after most scored origins - phase 43a measured 93.9% window
|
| 2913 |
+
coverage). The building axis is deliberately not validated (user judgment:
|
| 2914 |
+
same battery type across buildings).
|
| 2915 |
+
2. **Sole validator** (user, verbatim intent): "we only use purge score going
|
| 2916 |
+
forward, nothing else to validate a new model or optimization method."
|
| 2917 |
+
Documented as standing rule. LOBO's chimera-plan critique (24 stitched
|
| 2918 |
+
calibrations, no per-building cost attribution, seams inflate the transfer
|
| 2919 |
+
penalty) was the user's decisive argument against building-holdout cost.
|
| 2920 |
+
3. **Consolidation**: cv.py now contains ONLY the purge harness +
|
| 2921 |
+
oofcand/oofreport. all48/static/screen/lobo/ablate/wf/self removed;
|
| 2922 |
+
recover from git 2000281. exp_x29/exp_x30 no longer run as-is.
|
| 2923 |
+
|
| 2924 |
+
**Champion on THE SCORE: 1972.0** (warmup 3560.3 / panel 1839.6 / nearend
|
| 2925 |
+
604.8). Level chain, never cross-compare: self-fit 1661.7 (floor on) <
|
| 2926 |
+
exact-LOBO 1877.3 < 5-fold-OOF 1913.3 < purge 1972.0.
|
| 2927 |
+
|
| 2928 |
+
Paired purge-vs-LOBO diagnosis (59's run; LOBO reproduced 1877.3 exactly
|
| 2929 |
+
before removal): the purge is HARSHER overall (+94.7) - entirely warm-up
|
| 2930 |
+
(3560.3 vs 2875.3: purging the calendar-local +/-42d window starves early
|
| 2931 |
+
scenarios more than building-holdout does) - and modestly softer on the panel
|
| 2932 |
+
(-18.1/origin; sched 8.9 vs 9.2 flat, recall 0.468 vs 0.461, rec12 0.576 vs
|
| 2933 |
+
0.509). Reading: the same-device-fate channel exists but is ~-18/origin for
|
| 2934 |
+
the LINEAR champion - an order of magnitude below the +134/+216 self-fit
|
| 2935 |
+
flattery - because the label-overlap purge removes the direct channel and
|
| 2936 |
+
logbag cannot memorize identity. The known, accepted residual risk: flexible
|
| 2937 |
+
challengers (rf/gbm/dynamics) are flattered by same-battery training; the
|
| 2938 |
+
receipts are phases 51-54 and they are not to be re-argued unprompted.
|
| 2939 |
+
|
| 2940 |
+
The leaderboard closures (appetite, cert-first, pooling) rest on public
|
| 2941 |
+
measurements and survive any CV change. Public best 3c7b883 at 1661.45;
|
| 2942 |
+
private selection recommendation unchanged.
|