Thibaut Forest Claude Fable 5 commited on
Commit
bb5a0c1
·
1 Parent(s): 6ea03da

Phase 28 public refutation: revert static appetite to 0.8, cliff boost off

Browse files

Public look 2026-08-20 (6ea03da): 2188.78 vs 1661.45 (3c7b883). First public
exposure of the whole static stack at once; component signature is pure
over-inclusion - battery_swap +68%, early +444, late +13 (zero recall gain).

- ALPHA_STATIC default 1.5 -> 0.8: phase 26's undercount premise is refuted
on the real held-out fleet; phase 20 (the probability scale transfers /
overcounts) was right. The half-fleet holdout manufactures the undercount
(~6 deaths/scenario makes inclusions look cheap; marginal exchange rate
0.46 late-saved per early-spent locally, ~0.03 on public, break-even 1.0)
and is hereby refuted as an appetite adjudicator - count decisions are
public-look-only now. It stays valid for structural questions.
- CLIFF_SCALE default 2.2 -> 1.0: the boost was only ever adjudicated on top
of the refuted 1.5 baseline, and the public look condemns it transitively
(every cliff device was included at 1.5 + boost and late did not move).
- Retained: static-fit machinery (24b, structural), phase-27 penalty wiring
and failure isolation. Ledger addendum in EXPERIMENTS.md phase 28.

Verified: rehearsal on fake held-out splits (static path, no boost lines,
swap counts at 0.8 appetite); all-48 Docker byte-exact 1851.0186805555559.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (2) hide show
  1. EXPERIMENTS.md +73 -0
  2. batteryswap_example/submit.py +48 -10
EXPERIMENTS.md CHANGED
@@ -1143,6 +1143,32 @@ target, censored-anchor pricing, exact-cost-model optimization, building
1143
  group moves (phase-7/22 search saturation), isotonic rejection, CDF lag
1144
  (they over-swap early; we under-swap).
1145
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1146
  ## Phase 29 - fresh-eyes slate: generative label model, arrival rank/features, planner add/drop, policy control (all rejected; label-noise floor quantified)
1147
 
1148
  Trigger: user-requested outside review ("maybe we went in the wrong direction
@@ -1352,6 +1378,53 @@ devices live LONGER (r = +0.20), opposite the self-discharge story -
1352
  consistent with cold raising internal resistance and pulling the 2.40
1353
  crossing earlier.
1354
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1355
  ## Test-look ledger
1356
 
1357
  8 looks total (now historical - see protocol change above); look 9 = the
 
1143
  group moves (phase-7/22 search saturation), isotonic rejection, CDF lag
1144
  (they over-swap early; we under-swap).
1145
 
1146
+ **Public outcome (2026-08-20, submission 6ea03da): REFUTED - and it refutes
1147
+ phase 26 with it.** 2188.78 vs 1661.45 (3c7b883, still the live phase-23
1148
+ rolling config at alpha=0.8). This was the first public exposure of the
1149
+ ENTIRE static stack at once (24b static fit + 26 ALPHA_STATIC=1.5 + 27
1150
+ wiring + 28 cliff boost). Component signature: battery_swap 3.02 -> 5.08
1151
+ (+68% swaps), early 137 -> 582 (+444), ops +70, late 1212 -> 1225 (+13 -
1152
+ ZERO recall gain). That is pure over-inclusion: on the real held-out fleet
1153
+ the probability scale transfers (phase 20's finding: half-fleet sum(p42)
1154
+ 15.8 vs 9.1 true, it OVER-counts), and the half-fleet holdout construction
1155
+ that phase 26 and phase 28 both relied on manufactures the undercount their
1156
+ gains priced in. Late cost on public is saturated by voltage-silent deaths
1157
+ (the phases-1/6 result, again). Defaults reverted (ALPHA_STATIC=0.8,
1158
+ CLIFF_SCALE=1.0; static-fit machinery and phase-27 hardening retained).
1159
+ Standing lesson for every future phase: **the half-fleet holdout is refuted
1160
+ as an appetite adjudicator** - it disagreed with phase 20 and public sided
1161
+ with phase 20; appetite changes are public-look-only decisions now. WHY it
1162
+ mispredicts (parallel session, deployment-faithful sweep same day): the
1163
+ half-fleet scenarios carry ~6 deaths against public's much larger fleet, so
1164
+ added inclusions look cheaper than they are; the measured marginal exchange
1165
+ rate is 0.46 late-saved per early-spent at alpha 1.5->2.0 locally (0.21 at
1166
+ 2.0->2.5) and ~0.03 on public - all far below the 1.0 break-even. The
1167
+ half-fleet holdout remains valid for structural/mechanical questions
1168
+ (static-vs-rolling, failure isolation), not for anything count-related. The
1169
+ cliff boost's solo effect at alpha=0.8 is unproven (it was only ever
1170
+ measured on top of the refuted 1.5 baseline) and stays off by default.
1171
+
1172
  ## Phase 29 - fresh-eyes slate: generative label model, arrival rank/features, planner add/drop, policy control (all rejected; label-noise floor quantified)
1173
 
1174
  Trigger: user-requested outside review ("maybe we went in the wrong direction
 
1378
  consistent with cold raising internal resistance and pulling the 2.40
1379
  crossing earlier.
1380
 
1381
+ ## Phase 32 - "is p42 dangerous?" - window-portability bug FIXED, miss-cost ranking rejected
1382
+
1383
+ Two user hypotheses about p42, both tested; one found a real catastrophic-tail
1384
+ bug, the other closed a channel.
1385
+
1386
+ **(a) Window portability.** Worry: public/private could use a window other
1387
+ than 42d. The evaluator honours `planning_window_days` per scenario, so this
1388
+ is directly measurable: score unchanged plans against 28/56/84d evaluators.
1389
+
1390
+ | evaluator window | before fix | after fix |
1391
+ |---|---|---|
1392
+ | 28d | 728.5 | 728.5 (inert) |
1393
+ | 42d | 1086.6 | 1086.6 (byte-identical) |
1394
+ | 56d | **23581.9** | **1688.0** |
1395
+ | 84d | (same failure mode) | 3283.1 |
1396
+
1397
+ The model side degrades gracefully; the danger was in the OUTPUT convention.
1398
+ plan_tsroute parks non-included batteries at day 43 and relies on the
1399
+ evaluator cutting rows past the window - with W > 42 the evaluator EXECUTES
1400
+ all ~200 parked rows as real swaps and charges early penalties on the whole
1401
+ fleet. **Fixed** (`repark_for_window`, applied in all three plan paths):
1402
+ parked rows move to `max(W, 42)+1` read from the live settings. Inert at
1403
+ W <= 42 by construction (all-48 checksum byte-exact). Residual rise at 56/84d
1404
+ is legitimate, not a bug: a longer window simply contains deaths a 42d-ahead
1405
+ planner never scheduled for.
1406
+
1407
+ **(b) Miss-cost inclusion ranking (rejected).** Hypothesis: late_swap is high
1408
+ because inclusion ranks by raw p42, which treats a day-5 death (~430 if
1409
+ missed) like a day-40 death (~80). Tested `rank_score = p42 x E[late penalty
1410
+ if parked]` on the static held-out path (same alpha=1.5 count, same planner,
1411
+ only the boundary ORDER changes), 3 fleet splits x 16 origins:
1412
+
1413
+ | run | p42 | miss-cost | delta |
1414
+ |---|---|---|---|
1415
+ | s0 | 925.9 | 894.9 | -31.0 |
1416
+ | s0 reversed | 778.6 | 815.1 | +36.4 |
1417
+ | s1 | 914.0 | 898.7 | -15.3 |
1418
+
1419
+ 13W/12L/23T - no unanimity, rejected under the phase-26 standard. The
1420
+ mechanism does what it claims where it wins (late 462->439, 524->504) but the
1421
+ gains are small and reverse on one split: **p42 and death-day are strongly
1422
+ correlated** (a confident p42 usually means already-deep decline, i.e. dying
1423
+ soon), so the boundary is already nearly cost-ordered and the differences
1424
+ trade on day-quantile noise. Conclusion for the record: the residual
1425
+ late_swap is NOT mis-ordering among visible deaths - it is the voltage-silent
1426
+ deaths (channel closed, phases 6-13) plus label-lag noise.
1427
+
1428
  ## Test-look ledger
1429
 
1430
  8 looks total (now historical - see protocol change above); look 9 = the
batteryswap_example/submit.py CHANGED
@@ -52,14 +52,16 @@ MODEL_KIND = 'logbag'
52
  # late saturates ~1100 regardless - the remaining misses are voltage-silent
53
  # deaths that no inclusion appetite can catch.
54
  ALPHA = float(os.environ.get('BATTERYSWAP_ALPHA', '0.8'))
55
- # Phase 26: on a held-out fleet the static foreign fit's sum(p42) undercounts
56
- # (holdout: 5.5 vs 8 true deaths), so the static path needs a larger
57
- # inclusion appetite. Holdout sweep (4 runs, 3 seeds + reversed fleets):
58
- # 1.3 beats 0.8 in all four; curve flattens from 1.5 (0.8: 1156 -> 1.5: 916).
59
- # 1.5 takes ~93% of the gain without the plateau's over-inclusion edge.
60
- # Only the static branch reads this; the rolling/native path keeps ALPHA=0.8
61
- # where the panel decisively rejected higher appetites (phases 1/6).
62
- ALPHA_STATIC = float(os.environ.get('BATTERYSWAP_ALPHA_STATIC', '1.5'))
 
 
63
  # Break-even top-up (phase 11): additionally include every FRESH battery
64
  # (imputed EOL beyond the window) whose best-day expected cost beats parking,
65
  # gated to fire only where a wrong inclusion costs <= EC_PMAX. Empty at
@@ -94,7 +96,13 @@ FLOOR_GATE = int(os.environ.get('BATTERYSWAP_FLOOR_GATE', '20'))
94
  # (alpha=2.0: -55 but sd 298 and worst +421; alpha=2.5 turns NEGATIVE).
95
  # The native rolling path rejected it (panel +27): static branch ONLY.
96
  # CLIFF_SCALE=1.0 disables (boost becomes the identity).
97
- CLIFF_SCALE = float(os.environ.get('BATTERYSWAP_CLIFF_SCALE', '2.2'))
 
 
 
 
 
 
98
  CLIFF_FRAC = 0.35
99
  CLIFF_V = 2.46
100
  CLIFF_MIN_N = 20
@@ -317,6 +325,7 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
317
  plan = plan_tsroute(pred, None, locs, T, 0.05,
318
  travel_costs=scenario['travel_costs'],
319
  settings=scenario['settings'])
 
320
  plan['split'] = split
321
  plan['scenario'] = scenario['name']
322
  plans.append(plan)
@@ -325,6 +334,30 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
325
  return plans
326
 
327
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
328
  def fallback_split_plans(split, split_path, log=print):
329
  log(f' FALLBACK to static pickle for split={split}')
330
  with open(Path(__file__).parent / 'planners' / 'best.pickle', 'rb') as f:
@@ -334,6 +367,8 @@ def fallback_split_plans(split, split_path, log=print):
334
  for scenario, locs, cut, eol in iterate_scenarios(
335
  locations, timeseries, eol_times, scenarios):
336
  plan = planner.plan(cut, locs, scenario['travel_costs'], scenario['settings'])
 
 
337
  plan['split'] = split
338
  plan['scenario'] = scenario['name']
339
  plans.append(plan)
@@ -357,8 +392,11 @@ def last_resort_split_plans(split, split_path, log=print):
357
  T = pd.Timestamp(s['start_time'])
358
  e = eol.reindex(devices['device_id'])
359
  alive = devices['device_id'][e.isna().values | (e > T).values]
 
 
 
360
  plans.append(pd.DataFrame({
361
- 'day': (T + pd.Timedelta(days=WINDOW_DAYS + 1)).normalize(),
362
  'battery': alive.values,
363
  'split': split,
364
  'scenario': s['name'],
 
52
  # late saturates ~1100 regardless - the remaining misses are voltage-silent
53
  # deaths that no inclusion appetite can catch.
54
  ALPHA = float(os.environ.get('BATTERYSWAP_ALPHA', '0.8'))
55
+ # Phase 26 adopted ALPHA_STATIC=1.5 from the half-fleet holdout ("the static
56
+ # foreign fit undercounts sum(p42)"). PUBLIC REFUTED IT (2026-08-20,
57
+ # submission 6ea03da: 2188.78 vs 1661.45, battery_swap +68%, early +444,
58
+ # late +13 - pure over-inclusion, zero recall gain): on the real held-out
59
+ # fleet the probability scale transfers, exactly as the phase-20 fleet
60
+ # holdout found (sum(p42) 15.8 vs 9.1 true - if anything it OVER-counts).
61
+ # The half-fleet construction manufactures the undercount; do not trust it
62
+ # for appetite decisions. Late cost is saturated by voltage-silent deaths
63
+ # that no inclusion appetite can catch (phases 1/6). Default back to 0.8.
64
+ ALPHA_STATIC = float(os.environ.get('BATTERYSWAP_ALPHA_STATIC', '0.8'))
65
  # Break-even top-up (phase 11): additionally include every FRESH battery
66
  # (imputed EOL beyond the window) whose best-day expected cost beats parking,
67
  # gated to fire only where a wrong inclusion costs <= EC_PMAX. Empty at
 
96
  # (alpha=2.0: -55 but sd 298 and worst +421; alpha=2.5 turns NEGATIVE).
97
  # The native rolling path rejected it (panel +27): static branch ONLY.
98
  # CLIFF_SCALE=1.0 disables (boost becomes the identity).
99
+ # PUBLIC (2026-08-20, 6ea03da at ALPHA_STATIC=1.5 + CLIFF_SCALE=2.2):
100
+ # 2188.78 - the stacked appetite over-included with zero recall gain. The
101
+ # cliff boost was adjudicated ON TOP of the 1.5 baseline whose own premise
102
+ # public refuted, so its holdout gain is unproven at appetite 0.8 on the
103
+ # real fleet. Disabled by default until (if ever) a public look at
104
+ # ALPHA_STATIC=0.8 + CLIFF_SCALE=2.2 isolates its solo effect.
105
+ CLIFF_SCALE = float(os.environ.get('BATTERYSWAP_CLIFF_SCALE', '1.0'))
106
  CLIFF_FRAC = 0.35
107
  CLIFF_V = 2.46
108
  CLIFF_MIN_N = 20
 
325
  plan = plan_tsroute(pred, None, locs, T, 0.05,
326
  travel_costs=scenario['travel_costs'],
327
  settings=scenario['settings'])
328
+ plan = repark_for_window(plan, T, scenario['settings'], log)
329
  plan['split'] = split
330
  plan['scenario'] = scenario['name']
331
  plans.append(plan)
 
334
  return plans
335
 
336
 
337
+ def repark_for_window(plan, T, settings, log=print):
338
+ """Move parked rows past the ACTUAL window end.
339
+
340
+ plan_tsroute (and the pickled planner) park non-included batteries at
341
+ day 43, relying on the evaluator cutting rows past the window. That
342
+ convention hard-wires 42 into the OUTPUT: with planning_window_days > 42
343
+ the evaluator would EXECUTE every parked row as a real swap and charge
344
+ early penalties on the whole fleet (measured on a 56d evaluator: 23582
345
+ vs 1087 - a 20x blow-up). Inert at W <= 42, where day 43 is already
346
+ outside the window (checksum-verified byte-identical).
347
+ """
348
+ W = int(settings.planning_window_days)
349
+ if W <= WINDOW_DAYS:
350
+ return plan
351
+ parked_day = (T + pd.Timedelta(days=WINDOW_DAYS + 1)).normalize()
352
+ mask = plan['day'] == parked_day
353
+ if not mask.any():
354
+ return plan
355
+ log(f' window={W}d > {WINDOW_DAYS}d: re-parking {int(mask.sum())} rows')
356
+ plan = plan.copy()
357
+ plan.loc[mask, 'day'] = (T + pd.Timedelta(days=W + 1)).normalize()
358
+ return plan.sort_values('day', kind='stable').reset_index(drop=True)
359
+
360
+
361
  def fallback_split_plans(split, split_path, log=print):
362
  log(f' FALLBACK to static pickle for split={split}')
363
  with open(Path(__file__).parent / 'planners' / 'best.pickle', 'rb') as f:
 
367
  for scenario, locs, cut, eol in iterate_scenarios(
368
  locations, timeseries, eol_times, scenarios):
369
  plan = planner.plan(cut, locs, scenario['travel_costs'], scenario['settings'])
370
+ plan = repark_for_window(plan, pd.Timestamp(scenario['start_time']),
371
+ scenario['settings'], log)
372
  plan['split'] = split
373
  plan['scenario'] = scenario['name']
374
  plans.append(plan)
 
392
  T = pd.Timestamp(s['start_time'])
393
  e = eol.reindex(devices['device_id'])
394
  alive = devices['device_id'][e.isna().values | (e > T).values]
395
+ W = max(int(s['settings']['planning_window_days'])
396
+ if isinstance(s.get('settings'), dict) else WINDOW_DAYS,
397
+ WINDOW_DAYS)
398
  plans.append(pd.DataFrame({
399
+ 'day': (T + pd.Timedelta(days=W + 1)).normalize(),
400
  'battery': alive.values,
401
  'split': split,
402
  'scenario': s['name'],