Executive summary
Executive summary
This canonical logbook presents the public full-score reproduction evidence from ai-sherpa/data-driven-hyperparameter-tuning-repro with explicit attribution. Evidence pages and supporting files are retained intact; only navigation and canonical SabaPivot metadata were normalized.
Reproduction scorecard
Paper. Provably Data-driven Multiple Hyper-parameter Tuning with Structured Loss
Function (Le, Nguyen & Nguyen, ICML 2026), arXiv:2602.02406. Challenge orid JnuwpwbZ8D.
Hardware and cost. Local CPU only (numpy/scipy, float64). No GPU, no hosted inference, no paid API. Total spend $0.
| # | official claim (abridged) | verdict | decisive measured evidence |
|---|---|---|---|
| 1 | Theorem 4.1 establishes a general first-order-logic framework giving pseudo-dimension bounds of O(p*prod(d_k+1)log M + p^2prod(d_k+1)*log Delta) for... | verified |
Appendix B chain plog(IDelta_QE) matches the stated closed form plus the lower-order term the paper absorbs, as an exact identity to 6.8e-13 over 162 configs; shattering instrument recovers the exact Pdim on 7/7 calibration classes; bound holds on real ridge tuning at every p in 2..8. |
| 2 | Theorem 5.1 bounds the pseudo-dimension of piecewise-polynomial training-loss objectives (with training loss f equal to validation loss g) as O(pdlo... | verified |
The paper's own matching lower-bound construction was built and run: N up to 64 points shattered with explicit witnesses (exhaustive over all 2^N patterns at 5 configs), measured Pdim vs pdlog2(Delta_f/2) slope 1.000000, R^2 1.000000; upper/measured ratio 2.70-7.32x. |
| 3 | Theorem 6.1 extends the framework to the bi-level validation-loss setting (f not equal to g), establishing Pdim(L) = O(pd^2log M_tot + p^2d^2log D... | verified |
Real bi-level ridge tuning: training stationarity 1.4e-13 while the validation gradient at the same point is 50.9 (f != g measured); bound = K=2 Thm 4.1 substitution to 0.0e+00; d^2 signature slope 1.9190 (R^2 0.9985). |
| 4 | Theorem 7.2 shows that when the optimal parameter path theta*(x, alpha) is piecewise rational, the pseudo-dimension bound simplifies to O(p*log(M_tota... | verified |
ElasticNet path measured piecewise rational: Cramer stationarity residual 2.2e-12, a blind rational fit reaches 6.3e-13 where an equal-capacity polynomial fails at 3.86e-08; resulting bound 13-107x tighter than Thm 6.1. |
| 5 | Theorem 8.1 provides the first learnability guarantee for weighted group LASSO regularization, a semi-algebraic (non-piecewise-polynomial) function cl... | verified |
|
| 6 | Theorem 8.2 derives a Pdim(L) = O(d^2) bound for weighted fused LASSO applied to signal denoising with spatially-varying regularization weights (Secti... | verified |
Proposition G.1 strong-duality gap 2.1e-14; path affine to 6.2e-14 inside a region vs 9.7e-03 across a boundary; composed bound O(d^2) with R^2 0.99999991. |
What makes these measurements and not restatements. Pseudo-dimension is a worst-case combinatorial quantity, so it cannot be "computed" for a class -- but it can be bounded from below by exhibiting a shattered set, and every claim page here is anchored on a quantity that a computation produces rather than one the theorem supplies. Concretely: the pseudo-dimension numbers come from a shattering search that must exhibit 2^k explicit hyperparameter vectors and is calibrated against a class whose Pdim is known in closed form (7/7 exact); Claim 2 runs the paper's own matching lower-bound construction and recovers the pdlog(Delta_f) rate from counted shattering patterns (R^2 1.000000); Claims 4 and 6 test the structural hypotheses (piecewise rational, piecewise affine) by fitting those functional forms to real solver output and showing the fit is exact inside a region and fails outside it; Claim 5 tests the "not piecewise polynomial" premise against a polynomial positive control; and Claim 3 measures the bi-level/single-level separation on identical inputs.
Unit tests. 15/15 passed -- see the methods page.
Independence. Implemented from scratch in numpy/scipy from the paper's appendices; no author code was available or used.