Thibaut Forest Claude Fable 5 commited on
Commit ·
43761b5
1
Parent(s): 60e7aaf
Phase 52 addendum: building split extended to the cost harness - new cv all48oof protocol (building-OOF p42, full-fleet planning, paired self arm); panel 1679.2 -> 1813.6 (+134.4/origin, 8W/26L), self-building fit flatters all48 by ~+134/origin; prices harness optimism, no selection change
Browse files- EXPERIMENTS.md +32 -0
- batteryswap_example/cv.py +99 -0
EXPERIMENTS.md
CHANGED
|
@@ -2688,6 +2688,38 @@ Standing rule upgrade (supersedes the blind-loop "group by battery" rule):
|
|
| 2688 |
grouping is necessary but not sufficient; it still leaks the environment/
|
| 2689 |
install-batch channel that phase 10/13 closed on the feature side.
|
| 2690 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2691 |
## Phase 50 - the forecast axis: fleet-learned conditional dynamics (construction and screens)
|
| 2692 |
|
| 2693 |
Directive (user, 2026-08-22): innovate on the FORECAST part; reinforced with
|
|
|
|
| 2688 |
grouping is necessary but not sufficient; it still leaks the environment/
|
| 2689 |
install-batch channel that phase 10/13 closed on the feature side.
|
| 2690 |
|
| 2691 |
+
### Phase 52 addendum - the building split extended to the COST harness (`all48oof`)
|
| 2692 |
+
|
| 2693 |
+
Directive (user, 2026-08-22): "building split not only for the AUC but for
|
| 2694 |
+
the scenario split as well." New protocol `cv all48oof`: the walk-forward
|
| 2695 |
+
cost harness with BUILDING-CLEAN predictions - each device's p42 at every
|
| 2696 |
+
origin comes from a champion-config logbag fitted on pairs from every
|
| 2697 |
+
building except its own (5 greedy death-balanced folds, OOF-stitched), then
|
| 2698 |
+
the FULL fleet is planned as usual (planning must stay full-fleet: the count
|
| 2699 |
+
rule scales with fleet size - the phase-26 static-gate lesson). Paired
|
| 2700 |
+
`self` arm through identical machinery, cold-start floor OFF in both, so
|
| 2701 |
+
the per-origin delta isolates the self-building fit effect. Read the DELTA,
|
| 2702 |
+
not the level. Validity check: the self arm reproduces the canonical panel
|
| 2703 |
+
1679.2 and near-end 604.8 exactly (floor only fires in warm-up).
|
| 2704 |
+
|
| 2705 |
+
| block | self | oof | delta/origin | oof-better W / self-better L |
|
| 2706 |
+
|---|---|---|---|---|
|
| 2707 |
+
| warmup s_0-7 | 4256.1 | 4180.0 | -76.1 | 6W / 2L |
|
| 2708 |
+
| panel s_8-41 | **1679.2** | **1813.6** | **+134.4** | 8W / 26L |
|
| 2709 |
+
| nearend s_42-47 | 604.8 | 757.3 | +152.5 | 1W / 5L |
|
| 2710 |
+
| ALL-48 | 1974.4 | 2075.9 | +101.6 | 15W / 33L |
|
| 2711 |
+
|
| 2712 |
+
Ranking moves with the cost: panel rec12 0.572 -> 0.538, recall 0.571 ->
|
| 2713 |
+
0.542, ALL-48 rec12 0.597 -> 0.560. Warm-up's negative delta is noise
|
| 2714 |
+
(floor off, 3-20 train deaths). Reading: **self-building fitting flatters
|
| 2715 |
+
the walk-forward by ~+134/origin on the panel**; the transfer-shaped cost
|
| 2716 |
+
estimate for the champion on foreign buildings is the oof column. This is
|
| 2717 |
+
the cost-denominated size of the contamination the phase-52 screen showed
|
| 2718 |
+
as AUC margins - it prices the OPTIMISM of the harness, not a defect of the
|
| 2719 |
+
champion (both arms use the same model family; there is no building-free
|
| 2720 |
+
challenger that closes the gap, per the phase-52 screen). No selection
|
| 2721 |
+
change. Results /tmp/cv_all48oof.csv, seed 0.
|
| 2722 |
+
|
| 2723 |
## Phase 50 - the forecast axis: fleet-learned conditional dynamics (construction and screens)
|
| 2724 |
|
| 2725 |
Directive (user, 2026-08-22): innovate on the FORECAST part; reinforced with
|
batteryswap_example/cv.py
CHANGED
|
@@ -560,6 +560,101 @@ def ablation_report(df, ns, label, ref='base'):
|
|
| 560 |
print(f' {v:13s} ' + ' '.join(cells))
|
| 561 |
|
| 562 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 563 |
def main():
|
| 564 |
proto = sys.argv[1] if len(sys.argv) > 1 else 'all48'
|
| 565 |
variant = sys.argv[2] if len(sys.argv) > 2 else 'base'
|
|
@@ -585,6 +680,10 @@ def main():
|
|
| 585 |
'an appetite/count decision (see module docstring).')
|
| 586 |
elif proto == 'screen':
|
| 587 |
screen()
|
|
|
|
|
|
|
|
|
|
|
|
|
| 588 |
elif proto == 'ablate':
|
| 589 |
which = sys.argv[2] if len(sys.argv) > 2 else 'all48'
|
| 590 |
df, ns = ablate(protocol=which)
|
|
|
|
| 560 |
print(f' {v:13s} ' + ' '.join(cells))
|
| 561 |
|
| 562 |
|
| 563 |
+
# --------------------------------------------------------------------------
|
| 564 |
+
# protocol 5: all48 twin with building-OOF predictions (phase 52 follow-up)
|
| 565 |
+
# --------------------------------------------------------------------------
|
| 566 |
+
|
| 567 |
+
def run_all48_oof(n_folds=5, seed=0, origins=range(48),
|
| 568 |
+
dataset_path=Path('dataset')):
|
| 569 |
+
"""The walk-forward cost harness with BUILDING-CLEAN predictions.
|
| 570 |
+
|
| 571 |
+
all48 itself is temporal, not fold-based, and its PLANNING must stay
|
| 572 |
+
full-fleet: the count rule scales with fleet size, so holding buildings
|
| 573 |
+
out of the plan would manufacture the same count bias the static gate
|
| 574 |
+
carries (phase 26). What CAN be building-grouped is the prediction: here
|
| 575 |
+
device d's p42 at every origin comes from a model fitted on training
|
| 576 |
+
pairs from every building EXCEPT d's (OOF stitching), then the FULL
|
| 577 |
+
fleet is planned exactly as usual. Deployment analogy: the scored fleets
|
| 578 |
+
are foreign buildings, so this measures the walk-forward cost when the
|
| 579 |
+
model can never read its own building's identity.
|
| 580 |
+
|
| 581 |
+
Both arms (oof and self-fit) run through THIS function's machinery with
|
| 582 |
+
the cold-start floor off, so the paired per-origin delta isolates the
|
| 583 |
+
self-fleet fit effect. Levels may differ slightly from canonical all48
|
| 584 |
+
(floor off, no warm-start state); read the DELTA, not the level.
|
| 585 |
+
"""
|
| 586 |
+
t = load_table(dataset_path)
|
| 587 |
+
mc.FEATURES = list(BASE_FEATURES)
|
| 588 |
+
ctx = mc.load_context(dataset_path)
|
| 589 |
+
dev_bldg = (pd.read_csv(dataset_path / 'train' / 'devices.csv')
|
| 590 |
+
.drop_duplicates('device_id')
|
| 591 |
+
.set_index('device_id')['building_id'])
|
| 592 |
+
closed = t[t['origin'] <= t['origin'].max() - pd.Timedelta(days=WINDOW_DAYS)]
|
| 593 |
+
bldg_fold = building_folds(closed, dev_bldg, n_folds, seed)
|
| 594 |
+
|
| 595 |
+
p = mc.plan_tsroute
|
| 596 |
+
p.alpha, p.p_be, p.floor_day, p.p_conf = CHAMPION['p_star'], 0.0, 0, 0.5
|
| 597 |
+
p.ec_margin, p.ec_pmax = CHAMPION['ec_margin'], CHAMPION['ec_pmax']
|
| 598 |
+
p.split_days = CHAMPION['split_days']
|
| 599 |
+
p.floor_scale, p.recent_deaths = 0.0, 0.0
|
| 600 |
+
|
| 601 |
+
recs = []
|
| 602 |
+
for idx in origins:
|
| 603 |
+
c = ctx[idx]
|
| 604 |
+
T = c['start_time']
|
| 605 |
+
tr = mc.training_pairs(t, T)
|
| 606 |
+
if tr['dies42'].sum() < 3:
|
| 607 |
+
continue
|
| 608 |
+
rows = (t[t['origin_idx'] == idx].set_index('device_id')
|
| 609 |
+
.reindex(c['locs']['battery']).dropna(subset=['v_last']))
|
| 610 |
+
if rows.empty:
|
| 611 |
+
continue
|
| 612 |
+
locs = c['locs'][c['locs']['battery'].isin(rows.index)]
|
| 613 |
+
eol = c['eol'].reindex(rows.index)
|
| 614 |
+
tr_fold = tr['device_id'].map(dev_bldg).map(bldg_fold)
|
| 615 |
+
row_fold = rows.index.to_series().map(dev_bldg).map(bldg_fold)
|
| 616 |
+
for arm in ('self', 'oof'):
|
| 617 |
+
if arm == 'self':
|
| 618 |
+
pred = mc.SklearnRUL(CHAMPION['model']).fit(tr).predict(rows)
|
| 619 |
+
else:
|
| 620 |
+
parts = []
|
| 621 |
+
for f in sorted(row_fold.dropna().unique()):
|
| 622 |
+
m = mc.SklearnRUL(CHAMPION['model']).fit(tr[tr_fold != f])
|
| 623 |
+
parts.append(m.predict(rows[(row_fold == f).values]))
|
| 624 |
+
pred = pd.concat(parts).reindex(rows.index)
|
| 625 |
+
plan = mc.plan_tsroute(pred, None, locs, T, CHAMPION['day_q'],
|
| 626 |
+
travel_costs=c['travel_costs'],
|
| 627 |
+
settings=c['settings'])
|
| 628 |
+
_, _, overall = evaluate_plan(plan, locs, c['travel_costs'],
|
| 629 |
+
c['settings'], eol_times=eol,
|
| 630 |
+
start_time=T, verbose=0)
|
| 631 |
+
rs = mc.rank_stats(pred, plan, eol, T)
|
| 632 |
+
recs.append(dict(overall, **rs, origin=idx, arm=arm, seed=seed))
|
| 633 |
+
a, b = recs[-2], recs[-1]
|
| 634 |
+
print(f"s_{idx}: self {a['total_cost']:7.1f} oof {b['total_cost']:7.1f} "
|
| 635 |
+
f"delta {b['total_cost'] - a['total_cost']:+7.1f}",
|
| 636 |
+
flush=True, file=sys.stderr)
|
| 637 |
+
return pd.DataFrame(recs)
|
| 638 |
+
|
| 639 |
+
|
| 640 |
+
def oof_report(df):
|
| 641 |
+
for arm in ('self', 'oof'):
|
| 642 |
+
scorecard(df[df['arm'] == arm], f'all48oof / arm={arm} / floor off')
|
| 643 |
+
s = df[df['arm'] == 'self'].set_index('origin')['total_cost']
|
| 644 |
+
o = df[df['arm'] == 'oof'].set_index('origin')['total_cost']
|
| 645 |
+
d = (o - s).dropna()
|
| 646 |
+
for name, sel in BLOCKS:
|
| 647 |
+
dd = d[sel(pd.Series(d.index, index=d.index))]
|
| 648 |
+
if dd.empty:
|
| 649 |
+
continue
|
| 650 |
+
w, l = int((dd < 0).sum()), int((dd > 0).sum())
|
| 651 |
+
print(f'{name:16s} oof-minus-self {dd.mean():+7.1f}/origin '
|
| 652 |
+
f'(sum {dd.sum():+8.1f}, oof-better {w}W / self-better {l}L)')
|
| 653 |
+
print('\nPositive delta = self-fleet fitting was flattering the '
|
| 654 |
+
'walk-forward by that much; the scored fleets are foreign '
|
| 655 |
+
'buildings, so the oof arm is the transfer-shaped cost estimate.')
|
| 656 |
+
|
| 657 |
+
|
| 658 |
def main():
|
| 659 |
proto = sys.argv[1] if len(sys.argv) > 1 else 'all48'
|
| 660 |
variant = sys.argv[2] if len(sys.argv) > 2 else 'base'
|
|
|
|
| 680 |
'an appetite/count decision (see module docstring).')
|
| 681 |
elif proto == 'screen':
|
| 682 |
screen()
|
| 683 |
+
elif proto == 'all48oof':
|
| 684 |
+
df = run_all48_oof()
|
| 685 |
+
df.to_csv('/tmp/cv_all48oof.csv', index=False)
|
| 686 |
+
oof_report(df)
|
| 687 |
elif proto == 'ablate':
|
| 688 |
which = sys.argv[2] if len(sys.argv) > 2 else 'all48'
|
| 689 |
df, ns = ablate(protocol=which)
|