File size: 4,914 Bytes
e4aaa05 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | # T4 stacking lane: learned meta-combiner over the 3-source order — REJECTED
Date: 2026-08-23. Lane: STACKING (T4 count-flat campaign).
Baseline: head config 60d65da, CV 1541.3 on bgkf (`/tmp/ref_head_1541.csv`,
parity re-verified through this lane's own driver). Champion order =
fixed-weight rank blend .50*rank(logbag vm7) + .30*rank(WienerFHT) +
.20*rank(gapHGB depth2 on FEATURES+vmin7_rank+GAP4).
## Question
Can a LEARNED fold-safe meta-combiner (stacked generalization) beat the
fixed .50/.30/.20 weights by knowing per-device WHEN to trust which source
(context: wnr_n, staleness, gap_ratio, vmin7_rank; disagreement stats)?
## Method
Driver `/tmp/c24/run.py` (diagnostic `/tmp/c24/diag.py`). Inside each outer
fold's fit(train): fold-train split by BUILDING into K=4 inner folds
(deaths-sorted snake, deterministic; the established contamination rule —
never by row); out-of-fold predictions of all 3 sources on every train row,
pct-ranked WITHIN origin_idx groups (the deployment-time analog: the test
fleet is ranked at one origin, ~5 buildings — inner-test is also ~5
buildings, so rank distributions match); meta fit on
[r1,r2,r3, spread, |ri-rj|, wnr_n, staleness, gap_ratio, vmin7_rank] ->
dies42; sources refit on full fold-train; meta combines their test ranks.
Order-only: measured via cv.run_bgkf(kind='logbag', variant='vm7',
blend=(arm,1.0), p_star=1.0, split_days=3, timing_weight=2.0) — blend w2=1.0
reassigns the primary's sorted p42 values onto the meta's order, so the
count machinery is untouched. Paired vs the 48 per-scenario reference
totals. cv.py/models_cv.py md5-pinned per run (race guard); no repo files
touched.
## Results (paired vs head 1541.3, 48 scenarios)
| arm | meta | SCORE | delta±SE | sig | W/L | wilcoxon | swaps |
|---|---|---|---|---|---|---|---|
| par (parity) | fixed .50/.30/.20 | 1541.3 | +0.0±0.0 | 0.4σ | 1W/1L | – | 2.27 |
| a_log | logistic, ranks+disagree+context | 1604.5 | +63.2±44.8 | 1.4σ | 19W/25L | p0.208 | 2.26 |
| b_hgb | HGB depth2, same features | 1680.5 | +139.2±61.6 | 2.3σ | 17W/31L | p0.020 | 2.28 |
| c_ranks | HGB depth2, 3 ranks only | 1613.2 | +71.9±46.5 | 1.5σ | 17W/26L | p0.113 | 2.30 |
| d_cvx | convex weights, simplex grid by OOF AUC | 1598.6 | +57.3±41.0 | 1.4σ | 21W/21L | p0.620 | 2.28 |
| e_logrk | logistic, 3 ranks only | 1619.1 | +77.8±44.0 | 1.8σ | 17W/24L | p0.257 | 2.26 |
| e_log50 | 0.50*champ + 0.50*rank(a_log) | 1594.8 | +53.4±27.5 | 1.9σ | 13W/26L | p0.065 | 2.26 |
| e_log75 | 0.75*champ + 0.25*rank(a_log) | 1557.8 | +16.5±19.6 | 0.8σ | 16W/11L | p0.838 | 2.27 |
All deltas positive = WORSE than head. Every learned arm loses; b_hgb is a
2.3σ significant LOSS. Swaps flat everywhere (2.26-2.30 vs 2.27) — the
mechanism is count-flat as designed; the losses are pure ordering damage.
## Why it fails (diagnostic, per outer fold)
Per-source inner-OOF AUCs: lb 0.932-0.944, wiener 0.919-0.947, gap
0.941-0.955 — gapHGB is the BEST dies42 classifier in every fold. So any
classification-trained meta over-weights it:
- d_cvx learned weights (lb, wn, gap): (.15,.45,.40), (.10,.25,.65),
(.05,.30,.65), (.30,.30,.40), (.25,.20,.55) — gap at 2-3x its champion
weight, logbag starved. The champion .50/.30/.20 is IN the grid and the
learned points beat it on inner-OOF AUC in all 5 folds (+0.001 to +0.008)
— yet cost says the opposite. Objective mismatch, not estimation noise.
- a_log coefficients: r3 (gap) largest in 4/5 folds; context features get
small/unstable coefficients (vmin7_rank negative only because it double-
counts with the sources) — context adds nothing (a_log ~ e_logrk ~ c_ranks).
- Dose-response is monotone: 25% meta +16.5, 50% +53.4, 100% +63.2. The
learned order contributes no positive increment at ANY mixing weight.
Root cause: dies42-AUC/logloss over the closed-pair corpus is dominated by
separating obvious survivors from obvious deaths, where gapHGB excels. Plan
cost is decided by the fine ordering near the swap-inclusion cutoff and its
timing interplay, where logbag's calibrated top ranks matter — exactly what
the fully-mapped cost simplex (.50/.30/.20 optimal) already encodes. A
fold-safe meta cannot see cost, so it reliably walks AWAY from the cost
optimum toward the classification optimum (~.15/.30/.55 region).
## Verdict
Tier: REJECTED, mechanism dead — do not adopt, no arm to escalate.
Stacked generalization over these 3 sources is structurally mispriced under
any label-classification objective; the fixed .50/.30/.20 cost-mapped
weights stand. Do NOT revisit with a different meta-model family (HGB,
logistic, constrained-convex, ranks-only, context, shrinkage all measured
here); the only untried fix — per-fold weights tuned on plan cost — is the
already-dead static-simplex search wearing fold noise.
Artifacts: /tmp/c24/run.py, /tmp/c24/diag.py, /tmp/c24/*.out,
/tmp/cv_bgkfcand_c24_*.csv.
|