Phase 28 public refutation: revert static appetite to 0.8, cliff boost off
Browse filesPublic look 2026-08-20 (6ea03da): 2188.78 vs 1661.45 (3c7b883). First public
exposure of the whole static stack at once; component signature is pure
over-inclusion - battery_swap +68%, early +444, late +13 (zero recall gain).
- ALPHA_STATIC default 1.5 -> 0.8: phase 26's undercount premise is refuted
on the real held-out fleet; phase 20 (the probability scale transfers /
overcounts) was right. The half-fleet holdout manufactures the undercount
(~6 deaths/scenario makes inclusions look cheap; marginal exchange rate
0.46 late-saved per early-spent locally, ~0.03 on public, break-even 1.0)
and is hereby refuted as an appetite adjudicator - count decisions are
public-look-only now. It stays valid for structural questions.
- CLIFF_SCALE default 2.2 -> 1.0: the boost was only ever adjudicated on top
of the refuted 1.5 baseline, and the public look condemns it transitively
(every cliff device was included at 1.5 + boost and late did not move).
- Retained: static-fit machinery (24b, structural), phase-27 penalty wiring
and failure isolation. Ledger addendum in EXPERIMENTS.md phase 28.
Verified: rehearsal on fake held-out splits (static path, no boost lines,
swap counts at 0.8 appetite); all-48 Docker byte-exact 1851.0186805555559.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EXPERIMENTS.md +73 -0
- batteryswap_example/submit.py +48 -10
|
@@ -1143,6 +1143,32 @@ target, censored-anchor pricing, exact-cost-model optimization, building
|
|
| 1143 |
group moves (phase-7/22 search saturation), isotonic rejection, CDF lag
|
| 1144 |
(they over-swap early; we under-swap).
|
| 1145 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1146 |
## Phase 29 - fresh-eyes slate: generative label model, arrival rank/features, planner add/drop, policy control (all rejected; label-noise floor quantified)
|
| 1147 |
|
| 1148 |
Trigger: user-requested outside review ("maybe we went in the wrong direction
|
|
@@ -1352,6 +1378,53 @@ devices live LONGER (r = +0.20), opposite the self-discharge story -
|
|
| 1352 |
consistent with cold raising internal resistance and pulling the 2.40
|
| 1353 |
crossing earlier.
|
| 1354 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1355 |
## Test-look ledger
|
| 1356 |
|
| 1357 |
8 looks total (now historical - see protocol change above); look 9 = the
|
|
|
|
| 1143 |
group moves (phase-7/22 search saturation), isotonic rejection, CDF lag
|
| 1144 |
(they over-swap early; we under-swap).
|
| 1145 |
|
| 1146 |
+
**Public outcome (2026-08-20, submission 6ea03da): REFUTED - and it refutes
|
| 1147 |
+
phase 26 with it.** 2188.78 vs 1661.45 (3c7b883, still the live phase-23
|
| 1148 |
+
rolling config at alpha=0.8). This was the first public exposure of the
|
| 1149 |
+
ENTIRE static stack at once (24b static fit + 26 ALPHA_STATIC=1.5 + 27
|
| 1150 |
+
wiring + 28 cliff boost). Component signature: battery_swap 3.02 -> 5.08
|
| 1151 |
+
(+68% swaps), early 137 -> 582 (+444), ops +70, late 1212 -> 1225 (+13 -
|
| 1152 |
+
ZERO recall gain). That is pure over-inclusion: on the real held-out fleet
|
| 1153 |
+
the probability scale transfers (phase 20's finding: half-fleet sum(p42)
|
| 1154 |
+
15.8 vs 9.1 true, it OVER-counts), and the half-fleet holdout construction
|
| 1155 |
+
that phase 26 and phase 28 both relied on manufactures the undercount their
|
| 1156 |
+
gains priced in. Late cost on public is saturated by voltage-silent deaths
|
| 1157 |
+
(the phases-1/6 result, again). Defaults reverted (ALPHA_STATIC=0.8,
|
| 1158 |
+
CLIFF_SCALE=1.0; static-fit machinery and phase-27 hardening retained).
|
| 1159 |
+
Standing lesson for every future phase: **the half-fleet holdout is refuted
|
| 1160 |
+
as an appetite adjudicator** - it disagreed with phase 20 and public sided
|
| 1161 |
+
with phase 20; appetite changes are public-look-only decisions now. WHY it
|
| 1162 |
+
mispredicts (parallel session, deployment-faithful sweep same day): the
|
| 1163 |
+
half-fleet scenarios carry ~6 deaths against public's much larger fleet, so
|
| 1164 |
+
added inclusions look cheaper than they are; the measured marginal exchange
|
| 1165 |
+
rate is 0.46 late-saved per early-spent at alpha 1.5->2.0 locally (0.21 at
|
| 1166 |
+
2.0->2.5) and ~0.03 on public - all far below the 1.0 break-even. The
|
| 1167 |
+
half-fleet holdout remains valid for structural/mechanical questions
|
| 1168 |
+
(static-vs-rolling, failure isolation), not for anything count-related. The
|
| 1169 |
+
cliff boost's solo effect at alpha=0.8 is unproven (it was only ever
|
| 1170 |
+
measured on top of the refuted 1.5 baseline) and stays off by default.
|
| 1171 |
+
|
| 1172 |
## Phase 29 - fresh-eyes slate: generative label model, arrival rank/features, planner add/drop, policy control (all rejected; label-noise floor quantified)
|
| 1173 |
|
| 1174 |
Trigger: user-requested outside review ("maybe we went in the wrong direction
|
|
|
|
| 1378 |
consistent with cold raising internal resistance and pulling the 2.40
|
| 1379 |
crossing earlier.
|
| 1380 |
|
| 1381 |
+
## Phase 32 - "is p42 dangerous?" - window-portability bug FIXED, miss-cost ranking rejected
|
| 1382 |
+
|
| 1383 |
+
Two user hypotheses about p42, both tested; one found a real catastrophic-tail
|
| 1384 |
+
bug, the other closed a channel.
|
| 1385 |
+
|
| 1386 |
+
**(a) Window portability.** Worry: public/private could use a window other
|
| 1387 |
+
than 42d. The evaluator honours `planning_window_days` per scenario, so this
|
| 1388 |
+
is directly measurable: score unchanged plans against 28/56/84d evaluators.
|
| 1389 |
+
|
| 1390 |
+
| evaluator window | before fix | after fix |
|
| 1391 |
+
|---|---|---|
|
| 1392 |
+
| 28d | 728.5 | 728.5 (inert) |
|
| 1393 |
+
| 42d | 1086.6 | 1086.6 (byte-identical) |
|
| 1394 |
+
| 56d | **23581.9** | **1688.0** |
|
| 1395 |
+
| 84d | (same failure mode) | 3283.1 |
|
| 1396 |
+
|
| 1397 |
+
The model side degrades gracefully; the danger was in the OUTPUT convention.
|
| 1398 |
+
plan_tsroute parks non-included batteries at day 43 and relies on the
|
| 1399 |
+
evaluator cutting rows past the window - with W > 42 the evaluator EXECUTES
|
| 1400 |
+
all ~200 parked rows as real swaps and charges early penalties on the whole
|
| 1401 |
+
fleet. **Fixed** (`repark_for_window`, applied in all three plan paths):
|
| 1402 |
+
parked rows move to `max(W, 42)+1` read from the live settings. Inert at
|
| 1403 |
+
W <= 42 by construction (all-48 checksum byte-exact). Residual rise at 56/84d
|
| 1404 |
+
is legitimate, not a bug: a longer window simply contains deaths a 42d-ahead
|
| 1405 |
+
planner never scheduled for.
|
| 1406 |
+
|
| 1407 |
+
**(b) Miss-cost inclusion ranking (rejected).** Hypothesis: late_swap is high
|
| 1408 |
+
because inclusion ranks by raw p42, which treats a day-5 death (~430 if
|
| 1409 |
+
missed) like a day-40 death (~80). Tested `rank_score = p42 x E[late penalty
|
| 1410 |
+
if parked]` on the static held-out path (same alpha=1.5 count, same planner,
|
| 1411 |
+
only the boundary ORDER changes), 3 fleet splits x 16 origins:
|
| 1412 |
+
|
| 1413 |
+
| run | p42 | miss-cost | delta |
|
| 1414 |
+
|---|---|---|---|
|
| 1415 |
+
| s0 | 925.9 | 894.9 | -31.0 |
|
| 1416 |
+
| s0 reversed | 778.6 | 815.1 | +36.4 |
|
| 1417 |
+
| s1 | 914.0 | 898.7 | -15.3 |
|
| 1418 |
+
|
| 1419 |
+
13W/12L/23T - no unanimity, rejected under the phase-26 standard. The
|
| 1420 |
+
mechanism does what it claims where it wins (late 462->439, 524->504) but the
|
| 1421 |
+
gains are small and reverse on one split: **p42 and death-day are strongly
|
| 1422 |
+
correlated** (a confident p42 usually means already-deep decline, i.e. dying
|
| 1423 |
+
soon), so the boundary is already nearly cost-ordered and the differences
|
| 1424 |
+
trade on day-quantile noise. Conclusion for the record: the residual
|
| 1425 |
+
late_swap is NOT mis-ordering among visible deaths - it is the voltage-silent
|
| 1426 |
+
deaths (channel closed, phases 6-13) plus label-lag noise.
|
| 1427 |
+
|
| 1428 |
## Test-look ledger
|
| 1429 |
|
| 1430 |
8 looks total (now historical - see protocol change above); look 9 = the
|
|
@@ -52,14 +52,16 @@ MODEL_KIND = 'logbag'
|
|
| 52 |
# late saturates ~1100 regardless - the remaining misses are voltage-silent
|
| 53 |
# deaths that no inclusion appetite can catch.
|
| 54 |
ALPHA = float(os.environ.get('BATTERYSWAP_ALPHA', '0.8'))
|
| 55 |
-
# Phase 26
|
| 56 |
-
#
|
| 57 |
-
#
|
| 58 |
-
#
|
| 59 |
-
#
|
| 60 |
-
#
|
| 61 |
-
#
|
| 62 |
-
|
|
|
|
|
|
|
| 63 |
# Break-even top-up (phase 11): additionally include every FRESH battery
|
| 64 |
# (imputed EOL beyond the window) whose best-day expected cost beats parking,
|
| 65 |
# gated to fire only where a wrong inclusion costs <= EC_PMAX. Empty at
|
|
@@ -94,7 +96,13 @@ FLOOR_GATE = int(os.environ.get('BATTERYSWAP_FLOOR_GATE', '20'))
|
|
| 94 |
# (alpha=2.0: -55 but sd 298 and worst +421; alpha=2.5 turns NEGATIVE).
|
| 95 |
# The native rolling path rejected it (panel +27): static branch ONLY.
|
| 96 |
# CLIFF_SCALE=1.0 disables (boost becomes the identity).
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
CLIFF_FRAC = 0.35
|
| 99 |
CLIFF_V = 2.46
|
| 100 |
CLIFF_MIN_N = 20
|
|
@@ -317,6 +325,7 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
|
|
| 317 |
plan = plan_tsroute(pred, None, locs, T, 0.05,
|
| 318 |
travel_costs=scenario['travel_costs'],
|
| 319 |
settings=scenario['settings'])
|
|
|
|
| 320 |
plan['split'] = split
|
| 321 |
plan['scenario'] = scenario['name']
|
| 322 |
plans.append(plan)
|
|
@@ -325,6 +334,30 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
|
|
| 325 |
return plans
|
| 326 |
|
| 327 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 328 |
def fallback_split_plans(split, split_path, log=print):
|
| 329 |
log(f' FALLBACK to static pickle for split={split}')
|
| 330 |
with open(Path(__file__).parent / 'planners' / 'best.pickle', 'rb') as f:
|
|
@@ -334,6 +367,8 @@ def fallback_split_plans(split, split_path, log=print):
|
|
| 334 |
for scenario, locs, cut, eol in iterate_scenarios(
|
| 335 |
locations, timeseries, eol_times, scenarios):
|
| 336 |
plan = planner.plan(cut, locs, scenario['travel_costs'], scenario['settings'])
|
|
|
|
|
|
|
| 337 |
plan['split'] = split
|
| 338 |
plan['scenario'] = scenario['name']
|
| 339 |
plans.append(plan)
|
|
@@ -357,8 +392,11 @@ def last_resort_split_plans(split, split_path, log=print):
|
|
| 357 |
T = pd.Timestamp(s['start_time'])
|
| 358 |
e = eol.reindex(devices['device_id'])
|
| 359 |
alive = devices['device_id'][e.isna().values | (e > T).values]
|
|
|
|
|
|
|
|
|
|
| 360 |
plans.append(pd.DataFrame({
|
| 361 |
-
'day': (T + pd.Timedelta(days=
|
| 362 |
'battery': alive.values,
|
| 363 |
'split': split,
|
| 364 |
'scenario': s['name'],
|
|
|
|
| 52 |
# late saturates ~1100 regardless - the remaining misses are voltage-silent
|
| 53 |
# deaths that no inclusion appetite can catch.
|
| 54 |
ALPHA = float(os.environ.get('BATTERYSWAP_ALPHA', '0.8'))
|
| 55 |
+
# Phase 26 adopted ALPHA_STATIC=1.5 from the half-fleet holdout ("the static
|
| 56 |
+
# foreign fit undercounts sum(p42)"). PUBLIC REFUTED IT (2026-08-20,
|
| 57 |
+
# submission 6ea03da: 2188.78 vs 1661.45, battery_swap +68%, early +444,
|
| 58 |
+
# late +13 - pure over-inclusion, zero recall gain): on the real held-out
|
| 59 |
+
# fleet the probability scale transfers, exactly as the phase-20 fleet
|
| 60 |
+
# holdout found (sum(p42) 15.8 vs 9.1 true - if anything it OVER-counts).
|
| 61 |
+
# The half-fleet construction manufactures the undercount; do not trust it
|
| 62 |
+
# for appetite decisions. Late cost is saturated by voltage-silent deaths
|
| 63 |
+
# that no inclusion appetite can catch (phases 1/6). Default back to 0.8.
|
| 64 |
+
ALPHA_STATIC = float(os.environ.get('BATTERYSWAP_ALPHA_STATIC', '0.8'))
|
| 65 |
# Break-even top-up (phase 11): additionally include every FRESH battery
|
| 66 |
# (imputed EOL beyond the window) whose best-day expected cost beats parking,
|
| 67 |
# gated to fire only where a wrong inclusion costs <= EC_PMAX. Empty at
|
|
|
|
| 96 |
# (alpha=2.0: -55 but sd 298 and worst +421; alpha=2.5 turns NEGATIVE).
|
| 97 |
# The native rolling path rejected it (panel +27): static branch ONLY.
|
| 98 |
# CLIFF_SCALE=1.0 disables (boost becomes the identity).
|
| 99 |
+
# PUBLIC (2026-08-20, 6ea03da at ALPHA_STATIC=1.5 + CLIFF_SCALE=2.2):
|
| 100 |
+
# 2188.78 - the stacked appetite over-included with zero recall gain. The
|
| 101 |
+
# cliff boost was adjudicated ON TOP of the 1.5 baseline whose own premise
|
| 102 |
+
# public refuted, so its holdout gain is unproven at appetite 0.8 on the
|
| 103 |
+
# real fleet. Disabled by default until (if ever) a public look at
|
| 104 |
+
# ALPHA_STATIC=0.8 + CLIFF_SCALE=2.2 isolates its solo effect.
|
| 105 |
+
CLIFF_SCALE = float(os.environ.get('BATTERYSWAP_CLIFF_SCALE', '1.0'))
|
| 106 |
CLIFF_FRAC = 0.35
|
| 107 |
CLIFF_V = 2.46
|
| 108 |
CLIFF_MIN_N = 20
|
|
|
|
| 325 |
plan = plan_tsroute(pred, None, locs, T, 0.05,
|
| 326 |
travel_costs=scenario['travel_costs'],
|
| 327 |
settings=scenario['settings'])
|
| 328 |
+
plan = repark_for_window(plan, T, scenario['settings'], log)
|
| 329 |
plan['split'] = split
|
| 330 |
plan['scenario'] = scenario['name']
|
| 331 |
plans.append(plan)
|
|
|
|
| 334 |
return plans
|
| 335 |
|
| 336 |
|
| 337 |
+
def repark_for_window(plan, T, settings, log=print):
|
| 338 |
+
"""Move parked rows past the ACTUAL window end.
|
| 339 |
+
|
| 340 |
+
plan_tsroute (and the pickled planner) park non-included batteries at
|
| 341 |
+
day 43, relying on the evaluator cutting rows past the window. That
|
| 342 |
+
convention hard-wires 42 into the OUTPUT: with planning_window_days > 42
|
| 343 |
+
the evaluator would EXECUTE every parked row as a real swap and charge
|
| 344 |
+
early penalties on the whole fleet (measured on a 56d evaluator: 23582
|
| 345 |
+
vs 1087 - a 20x blow-up). Inert at W <= 42, where day 43 is already
|
| 346 |
+
outside the window (checksum-verified byte-identical).
|
| 347 |
+
"""
|
| 348 |
+
W = int(settings.planning_window_days)
|
| 349 |
+
if W <= WINDOW_DAYS:
|
| 350 |
+
return plan
|
| 351 |
+
parked_day = (T + pd.Timedelta(days=WINDOW_DAYS + 1)).normalize()
|
| 352 |
+
mask = plan['day'] == parked_day
|
| 353 |
+
if not mask.any():
|
| 354 |
+
return plan
|
| 355 |
+
log(f' window={W}d > {WINDOW_DAYS}d: re-parking {int(mask.sum())} rows')
|
| 356 |
+
plan = plan.copy()
|
| 357 |
+
plan.loc[mask, 'day'] = (T + pd.Timedelta(days=W + 1)).normalize()
|
| 358 |
+
return plan.sort_values('day', kind='stable').reset_index(drop=True)
|
| 359 |
+
|
| 360 |
+
|
| 361 |
def fallback_split_plans(split, split_path, log=print):
|
| 362 |
log(f' FALLBACK to static pickle for split={split}')
|
| 363 |
with open(Path(__file__).parent / 'planners' / 'best.pickle', 'rb') as f:
|
|
|
|
| 367 |
for scenario, locs, cut, eol in iterate_scenarios(
|
| 368 |
locations, timeseries, eol_times, scenarios):
|
| 369 |
plan = planner.plan(cut, locs, scenario['travel_costs'], scenario['settings'])
|
| 370 |
+
plan = repark_for_window(plan, pd.Timestamp(scenario['start_time']),
|
| 371 |
+
scenario['settings'], log)
|
| 372 |
plan['split'] = split
|
| 373 |
plan['scenario'] = scenario['name']
|
| 374 |
plans.append(plan)
|
|
|
|
| 392 |
T = pd.Timestamp(s['start_time'])
|
| 393 |
e = eol.reindex(devices['device_id'])
|
| 394 |
alive = devices['device_id'][e.isna().values | (e > T).values]
|
| 395 |
+
W = max(int(s['settings']['planning_window_days'])
|
| 396 |
+
if isinstance(s.get('settings'), dict) else WINDOW_DAYS,
|
| 397 |
+
WINDOW_DAYS)
|
| 398 |
plans.append(pd.DataFrame({
|
| 399 |
+
'day': (T + pd.Timedelta(days=W + 1)).normalize(),
|
| 400 |
'battery': alive.values,
|
| 401 |
'split': split,
|
| 402 |
'scenario': s['name'],
|