pages/executive-summary/page.md

# Executive summary

**Paper.** *Conditional Coverage Diagnostics for Conformal Prediction*,
arXiv:2512.11779v1, OpenReview `vaApZm6MKM`.

**Result.** All six claims reproduce.

**Synthetic design.** X ~ Uniform([−1,1]⁸), y|x ~ N(f(x), s(x)²) with
f(x) = x₁ + x₂² and s(x) = 0.4 + |x₁| + 0.5|x₂|; split conformal at α = 0.1 with
a homoskedastic base model on 4,000 train / 3,000 calibration points. Because
the interval is symmetric and the noise Gaussian, the **true** conditional
coverage is available in closed form,

    p(x) = Φ((q + ŷ(x) − f(x))/s(x)) − Φ((−q + ŷ(x) − f(x))/s(x)),

so every estimate is scored against ground truth rather than against another
estimate. The **oracle** interval f(x) ± z₍₁−α/2₎ s(x) gives p(x) = 1 − α
exactly and is the negative control each metric must return ≈ 0 on.

## Two things worth knowing before reproducing this paper

1. **The Table 2 statistic normalises over sample size as well as method.**
   Taking the max over methods alone — the obvious reading — gives LightGBM
   ≈ 83% instead of 68.4% and inverts none of the ordering but none of the
   numbers match either. Including the number-of-test-samples axis, and
   clipping negative ERT to 0, reproduces the paper to
   0.36 pp on LightGBM and
   0.48 pp on PartitionWise.

2. **An untuned classifier makes the strictly proper ERTs go negative.** With a
   default boosted-tree configuration (max_iter = 200, no regularisation) the
   out-of-fold classifier loses to the constant 1−α predictor and this run
   measured L2-ERT = −0.0045 and KL-ERT = −0.0594 on a genuinely violating
   predictor. A shallow, strongly regularised model with early stopping instead
   recovers 90% of the
   true L1 deviation. That is the paper's own thesis — the classifier is what
   gives the metric its power — appearing as a reproduction hazard, and it is
   why claim 2 matters as much as claim 1.

## Headline numbers

| claim | quantity | measured | reference |
|---|---|---|---|
| 1 | L1-ERT, oracle interval | +0.00117 | 0 |
| 1 | L1-ERT, standard conformal | 0.06327 | true 0.07030 |
| 2 | LightGBM L1-ERT power | 68.04% | paper 68.4% |
| 2 | PartitionWise L1-ERT power | 37.82% | paper 38.3% |
| 3 | CovGap error vs true L1 at n=5,000 | 0.05500 | true 0.06944 |
| 5 | KL-ERT = KL₊ + KL₋ residual | 8.94e-08 | 0 |
| 6 | in-sample L1-ERT on an oracle predictor | 0.17670 | 0 |

Each is paired with a control that must break, and each does: the oracle
interval drives every ERT to ≈ 0, marginal coverage stays at
0.9003 in both scenarios and so cannot
distinguish them, and dropping cross-fitting manufactures a violation where
there is none.