Executive summary
Paper. Conditional Coverage Diagnostics for Conformal Prediction,
arXiv:2512.11779v1, OpenReview vaApZm6MKM.
Result. All six claims reproduce.
Synthetic design. X ~ Uniform([−1,1]⁸), y|x ~ N(f(x), s(x)²) with f(x) = x₁ + x₂² and s(x) = 0.4 + |x₁| + 0.5|x₂|; split conformal at α = 0.1 with a homoskedastic base model on 4,000 train / 3,000 calibration points. Because the interval is symmetric and the noise Gaussian, the true conditional coverage is available in closed form,
p(x) = Φ((q + ŷ(x) − f(x))/s(x)) − Φ((−q + ŷ(x) − f(x))/s(x)),
so every estimate is scored against ground truth rather than against another estimate. The oracle interval f(x) ± z₍₁−α/2₎ s(x) gives p(x) = 1 − α exactly and is the negative control each metric must return ≈ 0 on.
Two things worth knowing before reproducing this paper
The Table 2 statistic normalises over sample size as well as method. Taking the max over methods alone — the obvious reading — gives LightGBM ≈ 83% instead of 68.4% and inverts none of the ordering but none of the numbers match either. Including the number-of-test-samples axis, and clipping negative ERT to 0, reproduces the paper to 0.36 pp on LightGBM and 0.48 pp on PartitionWise.
An untuned classifier makes the strictly proper ERTs go negative. With a default boosted-tree configuration (max_iter = 200, no regularisation) the out-of-fold classifier loses to the constant 1−α predictor and this run measured L2-ERT = −0.0045 and KL-ERT = −0.0594 on a genuinely violating predictor. A shallow, strongly regularised model with early stopping instead recovers 90% of the true L1 deviation. That is the paper's own thesis — the classifier is what gives the metric its power — appearing as a reproduction hazard, and it is why claim 2 matters as much as claim 1.
Headline numbers
| claim | quantity | measured | reference |
|---|---|---|---|
| 1 | L1-ERT, oracle interval | +0.00117 | 0 |
| 1 | L1-ERT, standard conformal | 0.06327 | true 0.07030 |
| 2 | LightGBM L1-ERT power | 68.04% | paper 68.4% |
| 2 | PartitionWise L1-ERT power | 37.82% | paper 38.3% |
| 3 | CovGap error vs true L1 at n=5,000 | 0.05500 | true 0.06944 |
| 5 | KL-ERT = KL₊ + KL₋ residual | 8.94e-08 | 0 |
| 6 | in-sample L1-ERT on an oracle predictor | 0.17670 | 0 |
Each is paired with a control that must break, and each does: the oracle interval drives every ERT to ≈ 0, marginal coverage stays at 0.9003 in both scenarios and so cannot distinguish them, and dropping cross-fitting manufactures a violation where there is none.