Thibaut Forest Claude Fable 5 commited on
Commit
43761b5
·
1 Parent(s): 60e7aaf

Phase 52 addendum: building split extended to the cost harness - new cv all48oof protocol (building-OOF p42, full-fleet planning, paired self arm); panel 1679.2 -> 1813.6 (+134.4/origin, 8W/26L), self-building fit flatters all48 by ~+134/origin; prices harness optimism, no selection change

Browse files
Files changed (2) hide show
  1. EXPERIMENTS.md +32 -0
  2. batteryswap_example/cv.py +99 -0
EXPERIMENTS.md CHANGED
@@ -2688,6 +2688,38 @@ Standing rule upgrade (supersedes the blind-loop "group by battery" rule):
2688
  grouping is necessary but not sufficient; it still leaks the environment/
2689
  install-batch channel that phase 10/13 closed on the feature side.
2690
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2691
  ## Phase 50 - the forecast axis: fleet-learned conditional dynamics (construction and screens)
2692
 
2693
  Directive (user, 2026-08-22): innovate on the FORECAST part; reinforced with
 
2688
  grouping is necessary but not sufficient; it still leaks the environment/
2689
  install-batch channel that phase 10/13 closed on the feature side.
2690
 
2691
+ ### Phase 52 addendum - the building split extended to the COST harness (`all48oof`)
2692
+
2693
+ Directive (user, 2026-08-22): "building split not only for the AUC but for
2694
+ the scenario split as well." New protocol `cv all48oof`: the walk-forward
2695
+ cost harness with BUILDING-CLEAN predictions - each device's p42 at every
2696
+ origin comes from a champion-config logbag fitted on pairs from every
2697
+ building except its own (5 greedy death-balanced folds, OOF-stitched), then
2698
+ the FULL fleet is planned as usual (planning must stay full-fleet: the count
2699
+ rule scales with fleet size - the phase-26 static-gate lesson). Paired
2700
+ `self` arm through identical machinery, cold-start floor OFF in both, so
2701
+ the per-origin delta isolates the self-building fit effect. Read the DELTA,
2702
+ not the level. Validity check: the self arm reproduces the canonical panel
2703
+ 1679.2 and near-end 604.8 exactly (floor only fires in warm-up).
2704
+
2705
+ | block | self | oof | delta/origin | oof-better W / self-better L |
2706
+ |---|---|---|---|---|
2707
+ | warmup s_0-7 | 4256.1 | 4180.0 | -76.1 | 6W / 2L |
2708
+ | panel s_8-41 | **1679.2** | **1813.6** | **+134.4** | 8W / 26L |
2709
+ | nearend s_42-47 | 604.8 | 757.3 | +152.5 | 1W / 5L |
2710
+ | ALL-48 | 1974.4 | 2075.9 | +101.6 | 15W / 33L |
2711
+
2712
+ Ranking moves with the cost: panel rec12 0.572 -> 0.538, recall 0.571 ->
2713
+ 0.542, ALL-48 rec12 0.597 -> 0.560. Warm-up's negative delta is noise
2714
+ (floor off, 3-20 train deaths). Reading: **self-building fitting flatters
2715
+ the walk-forward by ~+134/origin on the panel**; the transfer-shaped cost
2716
+ estimate for the champion on foreign buildings is the oof column. This is
2717
+ the cost-denominated size of the contamination the phase-52 screen showed
2718
+ as AUC margins - it prices the OPTIMISM of the harness, not a defect of the
2719
+ champion (both arms use the same model family; there is no building-free
2720
+ challenger that closes the gap, per the phase-52 screen). No selection
2721
+ change. Results /tmp/cv_all48oof.csv, seed 0.
2722
+
2723
  ## Phase 50 - the forecast axis: fleet-learned conditional dynamics (construction and screens)
2724
 
2725
  Directive (user, 2026-08-22): innovate on the FORECAST part; reinforced with
batteryswap_example/cv.py CHANGED
@@ -560,6 +560,101 @@ def ablation_report(df, ns, label, ref='base'):
560
  print(f' {v:13s} ' + ' '.join(cells))
561
 
562
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
563
  def main():
564
  proto = sys.argv[1] if len(sys.argv) > 1 else 'all48'
565
  variant = sys.argv[2] if len(sys.argv) > 2 else 'base'
@@ -585,6 +680,10 @@ def main():
585
  'an appetite/count decision (see module docstring).')
586
  elif proto == 'screen':
587
  screen()
 
 
 
 
588
  elif proto == 'ablate':
589
  which = sys.argv[2] if len(sys.argv) > 2 else 'all48'
590
  df, ns = ablate(protocol=which)
 
560
  print(f' {v:13s} ' + ' '.join(cells))
561
 
562
 
563
+ # --------------------------------------------------------------------------
564
+ # protocol 5: all48 twin with building-OOF predictions (phase 52 follow-up)
565
+ # --------------------------------------------------------------------------
566
+
567
+ def run_all48_oof(n_folds=5, seed=0, origins=range(48),
568
+ dataset_path=Path('dataset')):
569
+ """The walk-forward cost harness with BUILDING-CLEAN predictions.
570
+
571
+ all48 itself is temporal, not fold-based, and its PLANNING must stay
572
+ full-fleet: the count rule scales with fleet size, so holding buildings
573
+ out of the plan would manufacture the same count bias the static gate
574
+ carries (phase 26). What CAN be building-grouped is the prediction: here
575
+ device d's p42 at every origin comes from a model fitted on training
576
+ pairs from every building EXCEPT d's (OOF stitching), then the FULL
577
+ fleet is planned exactly as usual. Deployment analogy: the scored fleets
578
+ are foreign buildings, so this measures the walk-forward cost when the
579
+ model can never read its own building's identity.
580
+
581
+ Both arms (oof and self-fit) run through THIS function's machinery with
582
+ the cold-start floor off, so the paired per-origin delta isolates the
583
+ self-fleet fit effect. Levels may differ slightly from canonical all48
584
+ (floor off, no warm-start state); read the DELTA, not the level.
585
+ """
586
+ t = load_table(dataset_path)
587
+ mc.FEATURES = list(BASE_FEATURES)
588
+ ctx = mc.load_context(dataset_path)
589
+ dev_bldg = (pd.read_csv(dataset_path / 'train' / 'devices.csv')
590
+ .drop_duplicates('device_id')
591
+ .set_index('device_id')['building_id'])
592
+ closed = t[t['origin'] <= t['origin'].max() - pd.Timedelta(days=WINDOW_DAYS)]
593
+ bldg_fold = building_folds(closed, dev_bldg, n_folds, seed)
594
+
595
+ p = mc.plan_tsroute
596
+ p.alpha, p.p_be, p.floor_day, p.p_conf = CHAMPION['p_star'], 0.0, 0, 0.5
597
+ p.ec_margin, p.ec_pmax = CHAMPION['ec_margin'], CHAMPION['ec_pmax']
598
+ p.split_days = CHAMPION['split_days']
599
+ p.floor_scale, p.recent_deaths = 0.0, 0.0
600
+
601
+ recs = []
602
+ for idx in origins:
603
+ c = ctx[idx]
604
+ T = c['start_time']
605
+ tr = mc.training_pairs(t, T)
606
+ if tr['dies42'].sum() < 3:
607
+ continue
608
+ rows = (t[t['origin_idx'] == idx].set_index('device_id')
609
+ .reindex(c['locs']['battery']).dropna(subset=['v_last']))
610
+ if rows.empty:
611
+ continue
612
+ locs = c['locs'][c['locs']['battery'].isin(rows.index)]
613
+ eol = c['eol'].reindex(rows.index)
614
+ tr_fold = tr['device_id'].map(dev_bldg).map(bldg_fold)
615
+ row_fold = rows.index.to_series().map(dev_bldg).map(bldg_fold)
616
+ for arm in ('self', 'oof'):
617
+ if arm == 'self':
618
+ pred = mc.SklearnRUL(CHAMPION['model']).fit(tr).predict(rows)
619
+ else:
620
+ parts = []
621
+ for f in sorted(row_fold.dropna().unique()):
622
+ m = mc.SklearnRUL(CHAMPION['model']).fit(tr[tr_fold != f])
623
+ parts.append(m.predict(rows[(row_fold == f).values]))
624
+ pred = pd.concat(parts).reindex(rows.index)
625
+ plan = mc.plan_tsroute(pred, None, locs, T, CHAMPION['day_q'],
626
+ travel_costs=c['travel_costs'],
627
+ settings=c['settings'])
628
+ _, _, overall = evaluate_plan(plan, locs, c['travel_costs'],
629
+ c['settings'], eol_times=eol,
630
+ start_time=T, verbose=0)
631
+ rs = mc.rank_stats(pred, plan, eol, T)
632
+ recs.append(dict(overall, **rs, origin=idx, arm=arm, seed=seed))
633
+ a, b = recs[-2], recs[-1]
634
+ print(f"s_{idx}: self {a['total_cost']:7.1f} oof {b['total_cost']:7.1f} "
635
+ f"delta {b['total_cost'] - a['total_cost']:+7.1f}",
636
+ flush=True, file=sys.stderr)
637
+ return pd.DataFrame(recs)
638
+
639
+
640
+ def oof_report(df):
641
+ for arm in ('self', 'oof'):
642
+ scorecard(df[df['arm'] == arm], f'all48oof / arm={arm} / floor off')
643
+ s = df[df['arm'] == 'self'].set_index('origin')['total_cost']
644
+ o = df[df['arm'] == 'oof'].set_index('origin')['total_cost']
645
+ d = (o - s).dropna()
646
+ for name, sel in BLOCKS:
647
+ dd = d[sel(pd.Series(d.index, index=d.index))]
648
+ if dd.empty:
649
+ continue
650
+ w, l = int((dd < 0).sum()), int((dd > 0).sum())
651
+ print(f'{name:16s} oof-minus-self {dd.mean():+7.1f}/origin '
652
+ f'(sum {dd.sum():+8.1f}, oof-better {w}W / self-better {l}L)')
653
+ print('\nPositive delta = self-fleet fitting was flattering the '
654
+ 'walk-forward by that much; the scored fleets are foreign '
655
+ 'buildings, so the oof arm is the transfer-shaped cost estimate.')
656
+
657
+
658
  def main():
659
  proto = sys.argv[1] if len(sys.argv) > 1 else 'all48'
660
  variant = sys.argv[2] if len(sys.argv) > 2 else 'base'
 
680
  'an appetite/count decision (see module docstring).')
681
  elif proto == 'screen':
682
  screen()
683
+ elif proto == 'all48oof':
684
+ df = run_all48_oof()
685
+ df.to_csv('/tmp/cv_all48oof.csv', index=False)
686
+ oof_report(df)
687
  elif proto == 'ablate':
688
  which = sys.argv[2] if len(sys.argv) > 2 else 'all48'
689
  df, ns = ablate(protocol=which)