| # T4 stacking lane: learned meta-combiner over the 3-source order — REJECTED |
|
|
| Date: 2026-08-23. Lane: STACKING (T4 count-flat campaign). |
| Baseline: head config 60d65da, CV 1541.3 on bgkf (`/tmp/ref_head_1541.csv`, |
| parity re-verified through this lane's own driver). Champion order = |
| fixed-weight rank blend .50*rank(logbag vm7) + .30*rank(WienerFHT) + |
| .20*rank(gapHGB depth2 on FEATURES+vmin7_rank+GAP4). |
| |
| ## Question |
| |
| Can a LEARNED fold-safe meta-combiner (stacked generalization) beat the |
| fixed .50/.30/.20 weights by knowing per-device WHEN to trust which source |
| (context: wnr_n, staleness, gap_ratio, vmin7_rank; disagreement stats)? |
| |
| ## Method |
| |
| Driver `/tmp/c24/run.py` (diagnostic `/tmp/c24/diag.py`). Inside each outer |
| fold's fit(train): fold-train split by BUILDING into K=4 inner folds |
| (deaths-sorted snake, deterministic; the established contamination rule — |
| never by row); out-of-fold predictions of all 3 sources on every train row, |
| pct-ranked WITHIN origin_idx groups (the deployment-time analog: the test |
| fleet is ranked at one origin, ~5 buildings — inner-test is also ~5 |
| buildings, so rank distributions match); meta fit on |
| [r1,r2,r3, spread, |ri-rj|, wnr_n, staleness, gap_ratio, vmin7_rank] -> |
| dies42; sources refit on full fold-train; meta combines their test ranks. |
| Order-only: measured via cv.run_bgkf(kind='logbag', variant='vm7', |
| blend=(arm,1.0), p_star=1.0, split_days=3, timing_weight=2.0) — blend w2=1.0 |
| reassigns the primary's sorted p42 values onto the meta's order, so the |
| count machinery is untouched. Paired vs the 48 per-scenario reference |
| totals. cv.py/models_cv.py md5-pinned per run (race guard); no repo files |
| touched. |
| |
| ## Results (paired vs head 1541.3, 48 scenarios) |
| |
| | arm | meta | SCORE | delta±SE | sig | W/L | wilcoxon | swaps | |
| |---|---|---|---|---|---|---|---| |
| | par (parity) | fixed .50/.30/.20 | 1541.3 | +0.0±0.0 | 0.4σ | 1W/1L | – | 2.27 | |
| | a_log | logistic, ranks+disagree+context | 1604.5 | +63.2±44.8 | 1.4σ | 19W/25L | p0.208 | 2.26 | |
| | b_hgb | HGB depth2, same features | 1680.5 | +139.2±61.6 | 2.3σ | 17W/31L | p0.020 | 2.28 | |
| | c_ranks | HGB depth2, 3 ranks only | 1613.2 | +71.9±46.5 | 1.5σ | 17W/26L | p0.113 | 2.30 | |
| | d_cvx | convex weights, simplex grid by OOF AUC | 1598.6 | +57.3±41.0 | 1.4σ | 21W/21L | p0.620 | 2.28 | |
| | e_logrk | logistic, 3 ranks only | 1619.1 | +77.8±44.0 | 1.8σ | 17W/24L | p0.257 | 2.26 | |
| | e_log50 | 0.50*champ + 0.50*rank(a_log) | 1594.8 | +53.4±27.5 | 1.9σ | 13W/26L | p0.065 | 2.26 | |
| | e_log75 | 0.75*champ + 0.25*rank(a_log) | 1557.8 | +16.5±19.6 | 0.8σ | 16W/11L | p0.838 | 2.27 | |
| |
| All deltas positive = WORSE than head. Every learned arm loses; b_hgb is a |
| 2.3σ significant LOSS. Swaps flat everywhere (2.26-2.30 vs 2.27) — the |
| mechanism is count-flat as designed; the losses are pure ordering damage. |
| |
| ## Why it fails (diagnostic, per outer fold) |
| |
| Per-source inner-OOF AUCs: lb 0.932-0.944, wiener 0.919-0.947, gap |
| 0.941-0.955 — gapHGB is the BEST dies42 classifier in every fold. So any |
| classification-trained meta over-weights it: |
| |
| - d_cvx learned weights (lb, wn, gap): (.15,.45,.40), (.10,.25,.65), |
| (.05,.30,.65), (.30,.30,.40), (.25,.20,.55) — gap at 2-3x its champion |
| weight, logbag starved. The champion .50/.30/.20 is IN the grid and the |
| learned points beat it on inner-OOF AUC in all 5 folds (+0.001 to +0.008) |
| — yet cost says the opposite. Objective mismatch, not estimation noise. |
| - a_log coefficients: r3 (gap) largest in 4/5 folds; context features get |
| small/unstable coefficients (vmin7_rank negative only because it double- |
| counts with the sources) — context adds nothing (a_log ~ e_logrk ~ c_ranks). |
| - Dose-response is monotone: 25% meta +16.5, 50% +53.4, 100% +63.2. The |
| learned order contributes no positive increment at ANY mixing weight. |
| |
| Root cause: dies42-AUC/logloss over the closed-pair corpus is dominated by |
| separating obvious survivors from obvious deaths, where gapHGB excels. Plan |
| cost is decided by the fine ordering near the swap-inclusion cutoff and its |
| timing interplay, where logbag's calibrated top ranks matter — exactly what |
| the fully-mapped cost simplex (.50/.30/.20 optimal) already encodes. A |
| fold-safe meta cannot see cost, so it reliably walks AWAY from the cost |
| optimum toward the classification optimum (~.15/.30/.55 region). |
| |
| ## Verdict |
| |
| Tier: REJECTED, mechanism dead — do not adopt, no arm to escalate. |
| Stacked generalization over these 3 sources is structurally mispriced under |
| any label-classification objective; the fixed .50/.30/.20 cost-mapped |
| weights stand. Do NOT revisit with a different meta-model family (HGB, |
| logistic, constrained-convex, ranks-only, context, shrinkage all measured |
| here); the only untried fix — per-fold weights tuned on plan cost — is the |
| already-dead static-simplex search wearing fold noise. |
| |
| Artifacts: /tmp/c24/run.py, /tmp/c24/diag.py, /tmp/c24/*.out, |
| /tmp/cv_bgkfcand_c24_*.csv. |
| |
