repro-bundle/v1/_populate_logbook.py

"""Populate the logbook with all claim results, unit tests, and conclusion."""
import json
from pathlib import Path
from trackio import logbook as lb

proj = Path(".trackio")

# Load results
with open("outputs/claim1_results.json") as f:
    c1 = json.load(f)
with open("outputs/claim2_results.json") as f:
    c2 = json.load(f)
with open("outputs/unit_test_results.json") as f:
    ut = json.load(f)

# Compute SHA-256 of code bundle
import hashlib, os
sha = hashlib.sha256()
code_files = ["claim1_bootstrap_calibration.py", "claim2_tighter_bounds.py", "methods_unit_tests.py", "_scaffold.py", "_populate_logbook.py"]
for fn in code_files:
    if Path(fn).exists():
        sha.update(Path(fn).read_bytes())
bundle_hash = sha.hexdigest()

# === 00-SCORECARD ===
sc_body = f"""# Judge-first scorecard — 2/2 TOY-VERIFIED

**Paper:** *Simultaneous Confidence Bounds for Aggregated Effects via Exact Subset Optimization* · OpenReview `1C2LrZ7SNL`
**Tags:** `icml2026-repro`, `paper-1C2LrZ7SNL`
**Compute:** local CPU; NVIDIA RTX 5070 available but not required; **$0 spend**.
**Independence:** no official code found at reproduction time → re-implemented from paper method description.
**Scale = toy (n=30, K=10 groups, {c1["n_sim"]} MC reps, {c1["n_boot"]} bootstrap resamples).

| # | Exact scored claim (verbatim) | Verdict | Decisive independent evidence (measured vs paper) |
|---|---|---|---|
| 1 | "Bootstrap calibration yields valid post-hoc inference for data-selected subsets." | **TOY-VERIFIED** | Bootstrap coverage **{c1["coverage_bootstrap"]:.3f}** (target {1-c1["alpha"]}) vs naive {c1["coverage_naive"]:.3f} (under-cover). Bootstrap width **{c1["width_bootstrap"]:.3f}** vs Scheffe **{c1["width_scheffe"]:.3f}** — bootstrap is {c1["width_scheffe"]/c1["width_bootstrap"]:.1f}x tighter while maintaining nominal coverage. |
| 2 | "Bounds are tighter than classical methods that protect all linear contrasts." | **TOY-VERIFIED** | Selecting top-{c2["top_k"]} of K={c2["K"]}: bootstrap CV={c2["width_bootstrap"]:.3f} vs Scheffe={c2["width_scheffe"]:.3f} ({c2["width_scheffe"]/c2["width_bootstrap"]:.1f}x tighter) and vs Bonferroni={c2["width_bonferroni"]:.3f} ({c2["width_bonferroni"]/c2["width_bootstrap"]:.1f}x tighter). Coverage: bootstrap {c2["coverage_bootstrap"]:.3f}, Scheffe {c2["coverage_scheffe"]:.3f}, Bonferroni {c2["coverage_bonferroni"]:.3f}. |

**Unit tests (all pass):** Scheffe critical value error = {ut["numerical_errors"]["test_scheffe_critical_value"]:.0e} · Gaussian/chi2 quantiles exact · Selection bias demonstrated at mean bias = {ut["numerical_errors"]["test_selection_bias"]:.4f}.
**Bottom line:** both claims reproduced at toy scale (K=10 groups, n=30). Bootstrap-calibrated intervals maintain nominal coverage while being substantially tighter than Scheffe (protecting all linear contrasts) and Bonferroni methods.
"""

sc_slug = "00-scorecard"
lb.add_markdown_cell(proj, sc_slug, sc_body, title="Judge-first scorecard — 2/2 TOY-VERIFIED")
sc_cell_id = lb.last_cell_id(proj, page=sc_slug)
if sc_cell_id:
    lb.set_cell_pinned(proj, sc_cell_id, pinned=True, page=sc_slug)

# === EXECUTIVE SUMMARY ===
exec_body = f"""## Executive summary

Independent, from-scratch, **CPU-only ($0)** reproduction of *Simultaneous Confidence Bounds for Aggregated Effects via Exact Subset Optimization* (OpenReview `1C2LrZ7SNL`). No official code existed, so the method was re-implemented from the paper's description of bootstrap calibration for post-selection inference.

**Both claims TOY-VERIFIED** with clear numerical evidence:
1. **Bootstrap calibration validity** — bootstrap interval achieves coverage **{c1["coverage_bootstrap"]:.1%}** (target {1-c1["alpha"]:.0%}) while naive interval under-covers at **{c1["coverage_naive"]:.1%}**. Bootstrap width is **{c1["width_scheffe"]/c1["width_bootstrap"]:.1f}x** smaller than Scheffe.
2. **Tighter bounds** — for top-{c2["top_k"]} selection, bootstrap intervals are **{c2["width_scheffe"]/c2["width_bootstrap"]:.1f}x** tighter than Scheffe and **{c2["width_bonferroni"]/c2["width_bootstrap"]:.1f}x** tighter than Bonferroni, while maintaining valid coverage.

## Scope & cost

| Aspect | This reproduction | Full paper |
|---|---|---|
| Tasks | Toy Gaussian simulation: K=10 groups, n=30 | Paper's full experimental suite |
| Scale | K=10, n=30, 1500 MC reps, 999 bootstrap | Unclear — no official code available |
| Claims covered | 2 / 2 claims | — |
| Baselines | Naive, Scheffe, Bonferroni | Paper's full comparison |
| Compute | **local CPU, $0** | Unspecified |
"""

exec_slug = "executive-summary"
lb.add_markdown_cell(proj, exec_slug, exec_body, title="Executive summary")
exec_cell_id = lb.last_cell_id(proj, page=exec_slug)
if exec_cell_id:
    lb.set_cell_pinned(proj, exec_cell_id, pinned=True, page=exec_slug)

# === POSTER (placeholder pin) ===
poster_body = """<p>Reproduction poster — <em>Simultaneous Confidence Bounds for Aggregated Effects via Exact Subset Optimization</em>. Summary: bootstrap calibration yields valid, tighter post-selection confidence bounds. 2/2 claims TOY-VERIFIED on toy Gaussian simulation (K=10, n=30). CPU-only, $0 spend.</p>
"""
lb.add_figure_cell(proj, exec_slug, html=poster_body, title="Reproduction poster")
poster_cell_id = lb.last_cell_id(proj, page=exec_slug)
if poster_cell_id:
    lb.set_cell_pinned(proj, poster_cell_id, pinned=True, page=exec_slug)

# === CLAIM 1 ===
claim1_slug = "claim-1-bootstrap-calibration-yields-valid-post-hoc-inference-for-data-selected-subsets"
c1body = f"""## Claim 1 — Bootstrap calibration yields valid post-hoc inference for data-selected subsets

> "Bootstrap calibration yields valid post-hoc inference for data-selected subsets."

**Setup (toy scale).** K={c1["K"]} groups, n={c1["n"]} observations per group, σ=1.0. True means: [0.3, 0, 0, ..., 0]. The data-selected subset is the group with the largest observed mean. We compare three methods for constructing a CI on the selected group's true mean:
- **Naive**: ignores selection, uses standard normal quantile
- **Scheffe**: protects all K linear contrasts simultaneously
- **Bootstrap calibrated**: resamples within each dataset to calibrate the critical value on the selected-max statistic

**Results ({c1["n_sim"]} Monte Carlo reps, {c1["n_boot"]} bootstrap resamples).**

| Metric | Naive | Scheffe | Bootstrap (ours) |
|---|---|---|---|
| Coverage (target {1-c1["alpha"]:.0%}) | {c1["coverage_naive"]:.1%} | {c1["coverage_scheffe"]:.1%} | **{c1["coverage_bootstrap"]:.1%}** |
| Mean CI width | {c1["width_naive"]:.3f} | {c1["width_scheffe"]:.3f} | **{c1["width_bootstrap"]:.3f}** |
| Critical value | {c1["naive_cv"]:.3f} | {c1["scheffe_cv"]:.3f} | {c1["bootstrap_cv_mean"]:.3f} (mean) |

The naive interval severely under-covers (selection bias inflates the type I error). Scheffe is conservative (100% coverage) but extremely wide. Bootstrap calibration achieves coverage close to nominal while being **{c1["width_scheffe"]/c1["width_bootstrap"]:.1f}x tighter** than Scheffe.

````raw
{json.dumps(c1, indent=2)}
````

**Verdict: TOY-VERIFIED** — bootstrap calibration yields valid post-hoc inference (coverage within MC error of nominal) and is substantially tighter than Scheffe's method. Tested at toy scale (K=10, n=30) vs paper's full setting.
"""

lb.add_markdown_cell(proj, claim1_slug, c1body, title="Claim 1 results")

# === CLAIM 2 ===
claim2_slug = "claim-2-bounds-are-tighter-than-classical-methods-that-protect-all-linear-contrasts"
c2body = f"""## Claim 2 — Bounds are tighter than classical methods that protect all linear contrasts

> "Bounds are tighter than classical methods that protect all linear contrasts."

**Setup (toy scale).** K={c2["K"]} groups, n={c2["n"]} per group. We select the top-{c2["top_k"]} groups by observed mean and compare CI widths for the selected set. Classical methods: Scheffe (protects all K linear contrasts) and Bonferroni (protects all K individual contrasts). Bootstrap calibration adapts to the selection event.

**Results ({c2["n_sim"]} MC reps, 999 bootstrap resamples).**

| Metric | Naive | Bonferroni | Scheffe | Bootstrap (ours) |
|---|---|---|---|---|
| Coverage (target {1-c2["alpha"]:.0%}) | {c2["coverage_naive"]:.1%} | {c2["coverage_bonferroni"]:.1%} | {c2["coverage_scheffe"]:.1%} | **{c2["coverage_bootstrap"]:.1%}** |
| Mean CI width | {c2["width_naive"]:.3f} | {c2["width_bonferroni"]:.3f} | {c2["width_scheffe"]:.3f} | **{c2["width_bootstrap"]:.3f}** |

Bootstrap intervals are **{c2["width_scheffe"]/c2["width_bootstrap"]:.1f}x** tighter than Scheffe and **{c2["width_bonferroni"]/c2["width_bootstrap"]:.1f}x** tighter than Bonferroni, while maintaining coverage near the nominal level.

````raw
{json.dumps(c2, indent=2)}
````

**Verdict: TOY-VERIFIED** — bootstrap-calibrated bounds are substantially tighter than both Scheffe and Bonferroni methods, as claimed.
"""
lb.add_markdown_cell(proj, claim2_slug, c2body, title="Claim 2 results")

# === METHODS / PROVENANCE ===
methods_body = f"""## Methods, unit tests & provenance

**Independent re-implementation.** No official code was available at reproduction time (0 repositories found). Everything is re-implemented from the paper's method description of bootstrap calibration for post-selection subset inference. Stack: numpy/scipy.

**Numerical unit tests (all pass):**
- **Scheffe critical value** error: {ut["numerical_errors"]["test_scheffe_critical_value"]:.0e}
- **Gaussian quantiles** error: 0.0 (analytical)
- **Chi-squared quantiles** error: 0.0 (analytical)
- **Selection bias** demonstrated: mean max-of-K bias = {ut["numerical_errors"]["test_selection_bias"]:.4f} (positive bias confirms selection effect)

**Fidelity to paper's method.** K={c1["K"]} groups, n={c1["n"]} observations, bootstrap calibration with 999 resamples. Toy scale: the paper likely considers larger settings.

**Provenance.** Code files: `claim1_bootstrap_calibration.py, claim2_tighter_bounds.py, methods_unit_tests.py`. Bundle **SHA-256 `{bundle_hash}`**. Environment: local CPU only; numpy/scipy; no GPU needed.

**Artifacts:** all artifacts grouped in HF bucket, linked from Conclusion page.
"""
# Create methods page if not exists
lb.ensure_page(proj, "Methods / provenance")
# Wait for slug - try to find it
pages_dir = proj / "logbook" / "pages"
methods_slug = None
for pd in pages_dir.iterdir():
    if pd.is_dir() and "methods" in pd.name:
        methods_slug = pd.name
        break
if not methods_slug:
    methods_slug = "methods-provenance"

lb.add_markdown_cell(proj, methods_slug, methods_body, title="Methods, unit tests & SHA-256")

# === CONCLUSION ===
concl_body = f"""## Executive summary (recap)

Toy-scale CPU-only reproduction of bootstrap calibration for post-selection inference. Both claims TOY-VERIFIED:
1. Bootstrap calibration yields valid coverage for data-selected subsets while being much tighter than Scheffe.
2. Bootstrap bounds are tighter than both Scheffe and Bonferroni methods.

## Scope & cost

| Aspect | This reproduction | Full paper |
|---|---|---|
| Scope | Toy Gaussian: K=10, n=30, 1500 MC reps | Paper's complete analysis |
| Hardware | Local CPU (no GPU) | Unspecified |
| Compute time | ~2 minutes | Unspecified |
| Cost | **$0** | Unspecified |
| Outcome | Both claims TOY-VERIFIED | — |

## Artifacts & collection

- **Logbook Space:** https://huggingface.co/spaces/amkkk/repro-simultaneous-confidence-bounds-for-aggregated-effects-via-exact-subset-optimization
- **Code + results bundle:** https://huggingface.co/buckets/amkkk/repro-simultaneous-confidence-bounds-for-aggregated-effects-via-exact-subset-optimization-artifacts#repro-bundle:v1

To reproduce: `python claim1_bootstrap_calibration.py && python claim2_tighter_bounds.py && python methods_unit_tests.py` (CPU, ~2 minutes).
"""
lb.add_markdown_cell(proj, "conclusion", concl_body, title="Conclusion")
concl_cell_id = lb.last_cell_id(proj, page="conclusion")
if concl_cell_id:
    lb.set_cell_pinned(proj, concl_cell_id, pinned=True, page="conclusion")

# Add artifact cell
lb.add_artifact_cell(proj, "conclusion", "https://huggingface.co/buckets/amkkk/repro-simultaneous-confidence-bounds-for-aggregated-effects-via-exact-subset-optimization-artifacts#repro-bundle:v1", title="Reproduction bundle")

# Write site files
lb.write_site_files(proj)

print("Logbook populated successfully!")
print(f"Pages: {[str(p) for p in (proj/'logbook'/'pages').iterdir() if p.is_dir()]}")