Phase 37: dq_g20 cannot be made always-on - benefit and harm come from the same rows
Browse filesH1 (devw harm was a double-bootstrap artifact): refuted. Re-implemented with
true sample_weight in a manual bagging loop plus a uniform-weight control;
devw is +389 vs that control, clean dose-response, harm unchanged.
H2 (quar's 2.48 threshold is over-broad): refuted. It does drop 77 devices vs
the ~26 real ghosts, but tightening toward the ghosts is monotone in the wrong
direction on s_8-47 (2.40 +36.7 ... 2.48 +509.5) with the trend's limit being
'do not filter'. At 2.40 warm-up is identical to base, so the tight filter that
spares the panel also yields no cold-start gain.
Devices that look dead and survive are label noise at 3-20 positives and
informative hard negatives at 800+; the effect changes sign rather than
attenuating. Champion unchanged: all-48 1661.7, s_8-47 1518.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EXPERIMENTS.md +60 -0
|
@@ -1736,3 +1736,63 @@ window would surface it. It does not. The only negatives are the same
|
|
| 1736 |
tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
|
| 1737 |
s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
|
| 1738 |
noise). No hidden winner.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1736 |
tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to
|
| 1737 |
s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L,
|
| 1738 |
noise). No hidden winner.
|
| 1739 |
+
|
| 1740 |
+
## Phase 37 - can the phase-36 mechanism be made always-on? No.
|
| 1741 |
+
|
| 1742 |
+
dq_g20 is worth -49.8 on all-48 but never fires in deployment
|
| 1743 |
+
(static_uniq=82 vs gate 20). Two hypotheses for why the UNGATED version cost
|
| 1744 |
+
+374/+493 on the panel, both tested. `/tmp/always.py`.
|
| 1745 |
+
|
| 1746 |
+
### H1: devw's harm was a double-bootstrap artifact - REFUTED
|
| 1747 |
+
|
| 1748 |
+
The phase-36 `devw` resampled with replacement and then handed the result to
|
| 1749 |
+
BaggingClassifier, which bootstraps again. Re-implemented with true
|
| 1750 |
+
`sample_weight` inside a manual 15-bag loop, with a uniform-weight control
|
| 1751 |
+
(`mbag`) to absorb the implementation difference (that control is itself +90.5,
|
| 1752 |
+
so comparisons are made against IT, not against base):
|
| 1753 |
+
|
| 1754 |
+
| arm | vs base | **vs mbag control** |
|
| 1755 |
+
|---|---|---|
|
| 1756 |
+
| mbag (uniform) | +90.5 | - |
|
| 1757 |
+
| mbag_devw (1/n) | +479.7 | **+389** |
|
| 1758 |
+
| mbag_devsq (1/sqrt n) | +288.9 | +198 |
|
| 1759 |
+
| mbag_dev25 (1/n^0.25) | +91.0 | +0.5 |
|
| 1760 |
+
|
| 1761 |
+
Clean dose-response: the more device weighting, the worse. With the double
|
| 1762 |
+
bootstrap removed the harm is unchanged, so the implementation was not the
|
| 1763 |
+
cause. Weak weighting is simply inert.
|
| 1764 |
+
|
| 1765 |
+
### H2: quar's threshold (2.48) is over-broad - REFUTED
|
| 1766 |
+
|
| 1767 |
+
The filter is meant to remove the ~26 "ghost" trajectories (below the 2.45
|
| 1768 |
+
death threshold, no EOL recorded). At s_30 it actually drops:
|
| 1769 |
+
|
| 1770 |
+
| v_thr | rows | devices |
|
| 1771 |
+
|---|---|---|
|
| 1772 |
+
| 2.40 | 169 (0.3%) | **24** |
|
| 1773 |
+
| 2.45 | 642 (1.3%) | 53 |
|
| 1774 |
+
| 2.48 | 1066 (2.2%) | **77** |
|
| 1775 |
+
|
| 1776 |
+
So 2.48 removes 3x the genuine ghost count - mostly real slow-decliners that
|
| 1777 |
+
survived. Tightening it toward the ghosts should have kept the cold-start gain
|
| 1778 |
+
and dropped the panel harm. It does not. On s_8-47 (base 1518.0):
|
| 1779 |
+
|
| 1780 |
+
| v_thr | 2.40 | 2.44 | 2.45 | 2.46 | 2.47 | 2.48 | k=0.5 | k=0.8 |
|
| 1781 |
+
|---|---|---|---|---|---|---|---|---|
|
| 1782 |
+
| delta | **+36.7** | +163.0 | +181.3 | +198.9 | +458.8 | +509.5 | +138.6 | +78.3 |
|
| 1783 |
+
|
| 1784 |
+
Monotone, every value positive, and the limit of the trend is "do not filter"
|
| 1785 |
+
(base = 0). Softening (keeping 50%/80% of the poison rows instead of deleting
|
| 1786 |
+
them) walks the same curve back toward zero without crossing it. And at v_thr
|
| 1787 |
+
2.40 the warm-up block is 2379.7 - **identical to base** - i.e. the tight
|
| 1788 |
+
filter that spares the panel also delivers no cold-start gain.
|
| 1789 |
+
|
| 1790 |
+
### Conclusion
|
| 1791 |
+
|
| 1792 |
+
The benefit and the harm come from the SAME rows. Devices that look dead and
|
| 1793 |
+
survive are label noise when the model has 3-20 positives and informative hard
|
| 1794 |
+
negatives when it has 800+; the effect does not attenuate with data, it
|
| 1795 |
+
**changes sign**. No threshold, no soft-weighting and no sample-weight scheme
|
| 1796 |
+
separates the two. dq_g20 is intrinsically a cold-start mechanism and cannot be
|
| 1797 |
+
made always-on - which, given static_uniq=82, means it cannot be made to matter
|
| 1798 |
+
in deployment at all.
|