Thibaut Forest Claude Fable 5 commited on
Commit
9fd6de6
·
1 Parent(s): c64ca4a

Phase 37: dq_g20 cannot be made always-on - benefit and harm come from the same rows

Browse files

H1 (devw harm was a double-bootstrap artifact): refuted. Re-implemented with
true sample_weight in a manual bagging loop plus a uniform-weight control;
devw is +389 vs that control, clean dose-response, harm unchanged.

H2 (quar's 2.48 threshold is over-broad): refuted. It does drop 77 devices vs
the ~26 real ghosts, but tightening toward the ghosts is monotone in the wrong
direction on s_8-47 (2.40 +36.7 ... 2.48 +509.5) with the trend's limit being
'do not filter'. At 2.40 warm-up is identical to base, so the tight filter that
spares the panel also yields no cold-start gain.

Devices that look dead and survive are label noise at 3-20 positives and
informative hard negatives at 800+; the effect changes sign rather than
attenuating. Champion unchanged: all-48 1661.7, s_8-47 1518.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. EXPERIMENTS.md +60 -0
EXPERIMENTS.md CHANGED
@@ -1736,3 +1736,63 @@ window would surface it. It does not. The only negatives are the same
1736
  tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
1737
  s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
1738
  noise). No hidden winner.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1736
  tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
1737
  s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
1738
  noise). No hidden winner.
1739
+
1740
+ ## Phase 37 - can the phase-36 mechanism be made always-on? No.
1741
+
1742
+ dq_g20 is worth -49.8 on all-48 but never fires in deployment
1743
+ (static_uniq=82 vs gate 20). Two hypotheses for why the UNGATED version cost
1744
+ +374/+493 on the panel, both tested. `/tmp/always.py`.
1745
+
1746
+ ### H1: devw's harm was a double-bootstrap artifact - REFUTED
1747
+
1748
+ The phase-36 `devw` resampled with replacement and then handed the result to
1749
+ BaggingClassifier, which bootstraps again. Re-implemented with true
1750
+ `sample_weight` inside a manual 15-bag loop, with a uniform-weight control
1751
+ (`mbag`) to absorb the implementation difference (that control is itself +90.5,
1752
+ so comparisons are made against IT, not against base):
1753
+
1754
+ | arm | vs base | **vs mbag control** |
1755
+ |---|---|---|
1756
+ | mbag (uniform) | +90.5 | - |
1757
+ | mbag_devw (1/n) | +479.7 | **+389** |
1758
+ | mbag_devsq (1/sqrt n) | +288.9 | +198 |
1759
+ | mbag_dev25 (1/n^0.25) | +91.0 | +0.5 |
1760
+
1761
+ Clean dose-response: the more device weighting, the worse. With the double
1762
+ bootstrap removed the harm is unchanged, so the implementation was not the
1763
+ cause. Weak weighting is simply inert.
1764
+
1765
+ ### H2: quar's threshold (2.48) is over-broad - REFUTED
1766
+
1767
+ The filter is meant to remove the ~26 "ghost" trajectories (below the 2.45
1768
+ death threshold, no EOL recorded). At s_30 it actually drops:
1769
+
1770
+ | v_thr | rows | devices |
1771
+ |---|---|---|
1772
+ | 2.40 | 169 (0.3%) | **24** |
1773
+ | 2.45 | 642 (1.3%) | 53 |
1774
+ | 2.48 | 1066 (2.2%) | **77** |
1775
+
1776
+ So 2.48 removes 3x the genuine ghost count - mostly real slow-decliners that
1777
+ survived. Tightening it toward the ghosts should have kept the cold-start gain
1778
+ and dropped the panel harm. It does not. On s_8-47 (base 1518.0):
1779
+
1780
+ | v_thr | 2.40 | 2.44 | 2.45 | 2.46 | 2.47 | 2.48 | k=0.5 | k=0.8 |
1781
+ |---|---|---|---|---|---|---|---|---|
1782
+ | delta | **+36.7** | +163.0 | +181.3 | +198.9 | +458.8 | +509.5 | +138.6 | +78.3 |
1783
+
1784
+ Monotone, every value positive, and the limit of the trend is "do not filter"
1785
+ (base = 0). Softening (keeping 50%/80% of the poison rows instead of deleting
1786
+ them) walks the same curve back toward zero without crossing it. And at v_thr
1787
+ 2.40 the warm-up block is 2379.7 - **identical to base** - i.e. the tight
1788
+ filter that spares the panel also delivers no cold-start gain.
1789
+
1790
+ ### Conclusion
1791
+
1792
+ The benefit and the harm come from the SAME rows. Devices that look dead and
1793
+ survive are label noise when the model has 3-20 positives and informative hard
1794
+ negatives when it has 800+; the effect does not attenuate with data, it
1795
+ **changes sign**. No threshold, no soft-weighting and no sample-weight scheme
1796
+ separates the two. dq_g20 is intrinsically a cold-start mechanism and cannot be
1797
+ made always-on - which, given static_uniq=82, means it cannot be made to matter
1798
+ in deployment at all.