Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
eb395ee
·
1 Parent(s): bb5a0c1

Phase 33: late-swap root cause - count is optimal, ranking is the wall

Browse files

Forensics (panel s_8-41): late is 97.9% MISSES, 2.1% caught-late; we already
swap a median 12.2d before EOL, so no timing lever exists. Recall 176/308.
Missed deaths are LINGERERS (median days_below_245 63 vs 29.5 caught) sitting
at median rank 35 of ~450.

Definitive alpha curve on the SHIPPING rolling path (full evaluator cost):
0.6:1844 0.8:1679 1.0:1771 1.2:1862 1.5:2084 2.0:2369. Clean U with the
minimum at 0.8. More swaps DO cut late (1019->691) but early rises 3x faster
(444->1409); at 28.5 swaps our late still never reaches the leader's 499
while early is 2.3x theirs. No alpha reaches the leader profile - their 19
swaps are a consequence of better ranking (~40% marginal hit rate vs our 8%),
not a cause. Independently reproduces public 6ea03da (+8 swaps, late -13,
early +444).

Six ranking attacks rejected, all with the same signature - AUC UP, top-12
recall DOWN: zone features, v_corr_last+thresh_dist, RF ranking, rank blends,
cohort-restricted model, sub-daily/intra-day features. Plus miss-cost ranking
and prevalence recalibration (recorded deaths correlate NEGATIVELY with
next-window deaths). Sanity check: logbag top-12 175 vs best raw physics 119.

Methodology: score WHO candidates on top-12 recall, never AUC.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. EXPERIMENTS.md +71 -0
EXPERIMENTS.md CHANGED
@@ -1425,6 +1425,77 @@ trade on day-quantile noise. Conclusion for the record: the residual
1425
  late_swap is NOT mis-ordering among visible deaths - it is the voltage-silent
1426
  deaths (channel closed, phases 6-13) plus label-lag noise.
1427
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1428
  ## Test-look ledger
1429
 
1430
  8 looks total (now historical - see protocol change above); look 9 = the
 
1425
  late_swap is NOT mis-ordering among visible deaths - it is the voltage-silent
1426
  deaths (channel closed, phases 6-13) plus label-lag noise.
1427
 
1428
+ ## Phase 33 - late-swap root cause: the count is optimal, ranking is the wall
1429
+
1430
+ Public 6ea03da = 2188.78 (vs 3c7b883 1661.45) triggered a full attack on
1431
+ late_swap. Six mechanisms tested, all rejected; the diagnosis is now closed
1432
+ and quantitative.
1433
+
1434
+ ### Forensics (panel s_8-41, alpha=0.8, full evaluator)
1435
+ - late is **97.9% MISSES** (31070) vs 2.1% caught-but-swapped-late (660 over
1436
+ 34 origins). We already swap a median **12.2 days BEFORE** EOL, so the
1437
+ "shift swap days earlier" 20:1 lever DOES NOT EXIST.
1438
+ - recall 176/308 = 57%. Caught deaths: max rank 20, median rank 5.
1439
+ Missed deaths: **median rank 35** of ~450 - not marginal, not invisible.
1440
+ - miss signature = **LINGERERS**: median days_below_245 **63** vs 29.5 for
1441
+ caught; tau_projected 17.7 vs 6.4. They entered the 2.45 V zone long ago
1442
+ and decline slowly.
1443
+ - marginal economics: a marginal swap costs ~54 early and catches 8% x 251 =
1444
+ **EV 21**. Break-even density is 54/251 = **21.5%**. Ranks 11-12 run 32%
1445
+ (keep), 13-15 run 11.7%, 13-30 run 8% (reject).
1446
+
1447
+ ### The alpha question, settled on the SHIPPING path
1448
+ Definitive curve (rolling native fit, panel, full cost):
1449
+
1450
+ | alpha | 0.6 | **0.8** | 1.0 | 1.2 | 1.5 | 2.0 |
1451
+ |---|---|---|---|---|---|---|
1452
+ | swaps | 13.6 | **15.2** | 17.4 | 19.4 | 22.8 | 28.5 |
1453
+ | late | 1270 | **1019** | 944 | 866 | 817 | 691 |
1454
+ | early | 330 | **444** | 604 | 758 | 1017 | 1409 |
1455
+ | total | 1844 | **1679** | 1771 | 1862 | 2084 | 2369 |
1456
+
1457
+ Clean U with the minimum exactly at 0.8 (1.0 is better on 7 origins, worse on
1458
+ 24). More swaps DO cut misses - late falls 1019->691 - but early rises 3x
1459
+ faster. At 28.5 swaps our late (691) still never reaches the leader's 499
1460
+ while our early (1409) is 2.3x theirs: **there is no alpha that reaches the
1461
+ leader profile**. Their 19 swaps are a CONSEQUENCE of better ranking
1462
+ (marginal hit rate ~40% vs our 8%), not a cause. Matches the public result
1463
+ (+8 swaps -> late -13, early +444) and the 0.46 -> 0.21 -> 0.03 marginal
1464
+ rates from phase 28's addendum.
1465
+
1466
+ ### Ranking attacks - all rejected, all with the same signature
1467
+ Metric note (IMPORTANT): score WHO candidates on **top-12 recall**, never
1468
+ AUC. Every candidate below RAISED AUC while LOWERING the recall that
1469
+ converts to cost.
1470
+
1471
+ | candidate | AUC | top-12 recall (base 175/308) | verdict |
1472
+ |---|---|---|---|
1473
+ | zone features (days_below_245, in-zone, zone rate) | +0.0061 | 167 | rejected |
1474
+ | v_corr_last + thresh_dist (the label IS the 2.40 crossing) | +0.0088 | 173 | rejected |
1475
+ | both | +0.0111 | 169 | rejected |
1476
+ | RF ranking (rank_score) + logbag count | +0.0140 | 161 | rejected |
1477
+ | rank-avg 50/50, 75/25 logbag | +0.014 | 172 / 176 | noise (4W/3W) |
1478
+ | cohort model (train only on at-risk) | +0.0163 | 170 | rejected |
1479
+ | cohort + zone | +0.0185 | 169 | rejected |
1480
+ | sub-daily/intra-day (daily min, dip depth, dip accel, min-slope, dispersion) | +0.0009 | band hits 73->72 | rejected |
1481
+ | miss-cost ranking (p42 x E[late if parked]) | - | 13W/12L/23T | rejected |
1482
+ | prevalence recalibration (floor count by recently RECORDED deaths) | - | corr -0.03..-0.18 vs sum(p42) +0.39..+0.52 | rejected |
1483
+
1484
+ Cohort diagnostic (why the AUC/top-k divergence): the at-risk cohort is
1485
+ 28.9% of devices and holds **99% of deaths** (7.6% death rate inside vs 0.03%
1486
+ outside). A global model spends most capacity on the healthy-vs-sick boundary
1487
+ that AUC rewards and we never act on; but restricting to the cohort did NOT
1488
+ help either (170 vs 175), so the dilution hypothesis is refuted too.
1489
+
1490
+ Sanity check - ML ranking vs raw physics (top-12 of 308): logbag **175**,
1491
+ own-trajectory time-to-2.40 119, thresh_dist 108, v_corr_last 108,
1492
+ master-curve tau 51. Our ranking is far better than physics; it is simply not
1493
+ good enough at ranks 13-40 to make extra swaps pay.
1494
+
1495
+ **Conclusion**: late_swap is bounded by WHO-ranking quality among lingerers,
1496
+ not by appetite, timing, features-on-hand, or model family. The count channel
1497
+ is closed on the panel AND on the public leaderboard.
1498
+
1499
  ## Test-look ledger
1500
 
1501
  8 looks total (now historical - see protocol change above); look 9 = the