| <!doctype html> |
| <meta charset="utf-8"> |
| <title>Reproduction: Conditional Coverage Diagnostics for Conformal Prediction</title> |
| <style> |
| body{font:16px/1.6 system-ui,sans-serif;max-width:54rem;margin:3rem auto;padding:0 1.25rem;background:#0f1116;color:#e6e8ee} |
| a{color:#7aa2f7}code{background:#1a1d26;padding:.15em .4em;border-radius:4px} |
| th,td{border:1px solid #2a2f3a;padding:.4rem .6rem;text-align:left} |
| table{border-collapse:collapse;margin:1rem 0}h1{font-size:1.4rem} |
| .ok{color:#9ece6a} |
| </style> |
| <h1>Conditional Coverage Diagnostics for Conformal Prediction — reproduction</h1> |
| <p>arXiv:<a href="https://arxiv.org/abs/2512.11779">2512.11779</a>v1 · OpenReview <code>vaApZm6MKM</code></p> |
| <p>The synthetic design is chosen so the <em>true</em> conditional coverage |
| p(x) has a closed form, so every estimate is scored against ground truth |
| rather than against another estimate. The oracle interval gives p(x)=1−α |
| exactly and is the negative control each metric must return ≈0 on.</p> |
| <table> |
| <tr><th>Claim</th><th>Verdict</th><th>Headline</th></tr> |
| <tr><td>1 ERT family + constant-predictor floor</td><td class="ok">reproduced</td><td>oracle L1-ERT +0.0012; recovers 90% of true L1</td></tr> |
| <tr><td>2 LightGBM 68.4% vs PartitionWise 38.3%</td><td class="ok">reproduced</td><td>68.04% / 37.82% from released rows</td></tr> |
| <tr><td>3 ERT converges, CovGap does not</td><td class="ok">reproduced</td><td>CovGap error grows with n; ERT separates 14× better</td></tr> |
| <tr><td>4 Over/under-coverage decomposition</td><td class="ok">reproduced</td><td>60× ratio on a known-conservative predictor</td></tr> |
| <tr><td>5 Divergent KL± in classification</td><td class="ok">reproduced</td><td>residual 8.9e−8; 8/8 cells divergent</td></tr> |
| <tr><td>6 Algorithm 1 cross-fitting</td><td class="ok">reproduced</td><td>in-sample fabricates 0.177 on a valid predictor</td></tr> |
| </table> |
|
|
