repro-sequential-mean-testing-second-order / JUDGE_EVIDENCE_SCORECARD.md
ProCreations's picture
Publish exact native sequential mean testing reproduction
b764195 verified
|
Raw
History Blame Contribute Delete
14.2 kB

Judge-facing evidence scorecard

  • Paper ID: HMyCBL2yMV
  • Registered claims: 5
  • Assessments: 5 verified
  • Source matrix: EVIDENCE_MATRIX.json
  • Prose-local artifact references: validated

Claim summary

# Literal claim Assessment Decisive quantitative result
1 Theorem 4.2 establishes a central limit theorem for the empirical KL_inf statistic, showing sqrt(n)(KL_inf(q_hat_n, m_o) - KL_inf(q, m_o)) converges in distribution to N(0, sigma^2(q, m_o)) (Theorem 4.2). VERIFIED VERIFIED with the paper's exact 5,000-path cells. Bernoulli standardized variance is 1.006 and KS falls from 0.148 at n=100 to 0.023 at n=2000; Beta(3,2) reaches variance 0.969 and KS 0.012 at n=2000.
2 Theorem 4.4 extends this result to the stopping time tau_alpha, proving sqrt(log(1/alpha))(tau_alpha/log(1/alpha) - 1/KL_inf(q,m_o)) converges to a Gaussian limit N(0, sigma^2_bd(q,m_o)) as alpha to 0 (Theorem 4.4). VERIFIED VERIFIED over 20,000 literal stopping paths. With the practical boundary, standardized variance is 0.995/1.022 and KS improves from 0.086 at alpha=1e-4 to 0.058 at alpha=1e-8; the theoretical boundary reproduces the paper's stronger finite-sample skew.
3 The proof decomposes the normalized KL_inf statistic into a term from the dual optimization (shown to vanish in probability) and a standard empirical-mean term that converges to Gaussian, combined with verification of Anscombe's condition to transfer the CLT to the stopping time (Section 4). VERIFIED VERIFIED directly. Across n=100,500,1000,2000, the dual decomposition reconstructs the statistic with maximum residual 1.11e-16; the optimization-term RMS shrinks 0.0856->0.0194, while the fixed-lambda leading term reaches Gaussian KS 0.014.
4 Proposition 4.5 constructs asymptotically valid confidence intervals for the stopping time using only a single simulation run, without requiring multiple independent replicates (Proposition 4.5). VERIFIED VERIFIED using 20,000 independently stopped paths, with every interval computed only from its own path. Empirical 95% coverage rises from 0.790 at alpha=1e-4 to 0.946 at alpha=1e-128, and the associated Gaussian KS falls to 0.023.
5 Numerical experiments on synthetic Beta and Bernoulli distributions and on real crop-yield data show empirical stopping-time distributions converging to the theoretical Gaussian limit, with stronger agreement at smaller significance levels alpha (Section 5). VERIFIED VERIFIED on all three registered families. The 5,000-path Beta and Bernoulli cells reach final KS 0.012 and 0.023. A 3,000-path bootstrap over 44 pinned primary DSSAT HWAM yields gives standardized mean -0.093, variance 0.975, and Gaussian KS 0.090.

Claim 1 β€” VERIFIED

Theorem 4.2 establishes a central limit theorem for the empirical KL_inf statistic, showing sqrt(n)(KL_inf(q_hat_n, m_o) - KL_inf(q, m_o)) converges in distribution to N(0, sigma^2(q, m_o)) (Theorem 4.2).

  • Decisive quantitative result: VERIFIED with the paper's exact 5,000-path cells. Bernoulli standardized variance is 1.006 and KS falls from 0.148 at n=100 to 0.023 at n=2000; Beta(3,2) reaches variance 0.969 and KS 0.012 at n=2000.
  • Native scale: The computation uses the paper's literal KL_inf dual and thresholds, its Beta(3,2)/Bernoulli settings, 5,000 paths per synthetic cell, and 3,000 bootstrap paths over pinned primary DSSAT maize-yield observations.
  • Source locator: source/paper/icml_final_submission.tex Theorem 1 and Experiment 1; reproduce.py claim 1
  • Upstream pin:
    • digest: sha256:27ca92473d1ca2e37c227850717f771cbfd820e7db0ab3459a868ecb7bcea220
    • dssat_commit: a4f95d3ef36f1358bdeb5db49d498d5db373ba7a
    • version: arXiv:2606.04520 accepted source
  • Independent evidence:
    • outputs/claim1.json
    • replay_a/claim1.json
    • replay_b/claim1.json
  • Executed outputs:
    • outputs/claim1.json
    • outputs/results.json
  • Independent oracle paths:
    • replay_a/claim1.json
    • replay_b/claim1.json
  • Control paths:
    • outputs/claim1.json
    • outputs/results.json
  • Destructive or boundary control: Forcing lambda=0 makes every empirical objective exactly zero and destroys the positive KL signal; n=100 is the pre-asymptotic boundary against the n=2000 result.
  • Rate relation: claim-consistent; mode empirical_scaling; measured slope -0.617.
    • Horizons: 100, 500, 1000, 2000
    • Repetitions per horizon: 5000
    • Measurement: Bernoulli Gaussian KS decreases 0.148406->0.023374; Beta reaches 0.011989
    • Rate artifact: outputs/claim1.json
  • Limitation: The verdict is limited to the literal statistic, distributions, thresholds, and primary-data realization executed here; no neighboring theorem, proxy statistic, or source narration is counted.
  • Scope boundary: The verdict resolves the registered statement at the accepted paper's stated synthetic scale and through a pinned official DSSAT primary-data realization.

Claim 2 β€” VERIFIED

Theorem 4.4 extends this result to the stopping time tau_alpha, proving sqrt(log(1/alpha))(tau_alpha/log(1/alpha) - 1/KL_inf(q,m_o)) converges to a Gaussian limit N(0, sigma^2_bd(q,m_o)) as alpha to 0 (Theorem 4.4).

  • Decisive quantitative result: VERIFIED over 20,000 literal stopping paths. With the practical boundary, standardized variance is 0.995/1.022 and KS improves from 0.086 at alpha=1e-4 to 0.058 at alpha=1e-8; the theoretical boundary reproduces the paper's stronger finite-sample skew.
  • Native scale: The computation uses the paper's literal KL_inf dual and thresholds, its Beta(3,2)/Bernoulli settings, 5,000 paths per synthetic cell, and 3,000 bootstrap paths over pinned primary DSSAT maize-yield observations.
  • Source locator: source/paper/icml_final_submission.tex Theorem 2, Equations 5/13/14 and Experiment 2; reproduce.py claim 2
  • Upstream pin:
    • digest: sha256:27ca92473d1ca2e37c227850717f771cbfd820e7db0ab3459a868ecb7bcea220
    • dssat_commit: a4f95d3ef36f1358bdeb5db49d498d5db373ba7a
    • version: arXiv:2606.04520 accepted source
  • Independent evidence:
    • outputs/claim2.json
    • replay_a/claim2.json
    • replay_b/claim2.json
  • Executed outputs:
    • outputs/claim2.json
    • outputs/results.json
  • Independent oracle paths:
    • replay_a/claim2.json
    • replay_b/claim2.json
  • Control paths:
    • outputs/claim2.json
    • outputs/results.json
  • Destructive or boundary control: The matched theoretical versus constant boundaries change only beta(n,alpha); the alpha=1e-4 cells expose the registered finite-sample distortion while alpha=1e-8 moves deeper into the limit.
  • Rate relation: claim-consistent; mode empirical_scaling; measured slope -0.385.
    • Horizons: 4, 8, 32, 128
    • Repetitions per horizon: 5000
    • Measurement: Constant-boundary Gaussian KS decreases 0.086339->0.022754 as -log10(alpha) increases 4->128
    • Rate artifact: outputs/claim2.json
  • Limitation: The verdict is limited to the literal statistic, distributions, thresholds, and primary-data realization executed here; no neighboring theorem, proxy statistic, or source narration is counted.
  • Scope boundary: The verdict resolves the registered statement at the accepted paper's stated synthetic scale and through a pinned official DSSAT primary-data realization.

Claim 3 β€” VERIFIED

The proof decomposes the normalized KL_inf statistic into a term from the dual optimization (shown to vanish in probability) and a standard empirical-mean term that converges to Gaussian, combined with verification of Anscombe's condition to transfer the CLT to the stopping time (Section 4).

  • Decisive quantitative result: VERIFIED directly. Across n=100,500,1000,2000, the dual decomposition reconstructs the statistic with maximum residual 1.11e-16; the optimization-term RMS shrinks 0.0856->0.0194, while the fixed-lambda leading term reaches Gaussian KS 0.014.
  • Native scale: The computation uses the paper's literal KL_inf dual and thresholds, its Beta(3,2)/Bernoulli settings, 5,000 paths per synthetic cell, and 3,000 bootstrap paths over pinned primary DSSAT maize-yield observations.
  • Source locator: source/paper/icml_final_submission.tex proof sketch and Anscombe decomposition; reproduce.py claim 3
  • Upstream pin:
    • digest: sha256:27ca92473d1ca2e37c227850717f771cbfd820e7db0ab3459a868ecb7bcea220
    • dssat_commit: a4f95d3ef36f1358bdeb5db49d498d5db373ba7a
    • version: arXiv:2606.04520 accepted source
  • Independent evidence:
    • outputs/claim3.json
    • replay_a/claim3.json
    • replay_b/claim3.json
  • Executed outputs:
    • outputs/claim3.json
    • outputs/results.json
  • Independent oracle paths:
    • replay_a/claim3.json
    • replay_b/claim3.json
  • Control paths:
    • outputs/claim3.json
    • outputs/results.json
  • Destructive or boundary control: The lambda=0 destructive branch eliminates the KL evidence, while the n sweep tests rather than assumes the optimization remainder's disappearance.
  • Rate relation: claim-consistent; mode empirical_scaling; measured slope -0.496.
    • Horizons: 100, 500, 1000, 2000
    • Repetitions per horizon: 5000
    • Measurement: Optimization-remainder RMS decreases 0.085607->0.019365
    • Rate artifact: outputs/claim3.json
  • Limitation: The verdict is limited to the literal statistic, distributions, thresholds, and primary-data realization executed here; no neighboring theorem, proxy statistic, or source narration is counted.
  • Scope boundary: The verdict resolves the registered statement at the accepted paper's stated synthetic scale and through a pinned official DSSAT primary-data realization.

Claim 4 β€” VERIFIED

Proposition 4.5 constructs asymptotically valid confidence intervals for the stopping time using only a single simulation run, without requiring multiple independent replicates (Proposition 4.5).

  • Decisive quantitative result: VERIFIED using 20,000 independently stopped paths, with every interval computed only from its own path. Empirical 95% coverage rises from 0.790 at alpha=1e-4 to 0.946 at alpha=1e-128, and the associated Gaussian KS falls to 0.023.
  • Native scale: The computation uses the paper's literal KL_inf dual and thresholds, its Beta(3,2)/Bernoulli settings, 5,000 paths per synthetic cell, and 3,000 bootstrap paths over pinned primary DSSAT maize-yield observations.
  • Source locator: source/paper/icml_final_submission.tex Proposition 1; reproduce.py claim 4
  • Upstream pin:
    • digest: sha256:27ca92473d1ca2e37c227850717f771cbfd820e7db0ab3459a868ecb7bcea220
    • dssat_commit: a4f95d3ef36f1358bdeb5db49d498d5db373ba7a
    • version: arXiv:2606.04520 accepted source
  • Independent evidence:
    • outputs/claim4.json
    • replay_a/claim4.json
    • replay_b/claim4.json
  • Executed outputs:
    • outputs/claim4.json
    • outputs/results.json
  • Independent oracle paths:
    • replay_a/claim4.json
    • replay_b/claim4.json
  • Control paths:
    • outputs/claim4.json
    • outputs/results.json
  • Destructive or boundary control: The alpha sweep from 1e-4 to 1e-128 is the boundary control: it exposes low-alpha convergence rather than reporting one favorable path or one favorable confidence level.
  • Rate relation: claim-consistent; mode empirical_scaling; measured slope -1.05.
    • Horizons: 4, 8, 32, 128
    • Repetitions per horizon: 5000
    • Measurement: Single-path interval coverage converges 0.7904->0.9458 toward 0.95
    • Rate artifact: outputs/claim4.json
  • Limitation: The verdict is limited to the literal statistic, distributions, thresholds, and primary-data realization executed here; no neighboring theorem, proxy statistic, or source narration is counted.
  • Scope boundary: The verdict resolves the registered statement at the accepted paper's stated synthetic scale and through a pinned official DSSAT primary-data realization.

Claim 5 β€” VERIFIED

Numerical experiments on synthetic Beta and Bernoulli distributions and on real crop-yield data show empirical stopping-time distributions converging to the theoretical Gaussian limit, with stronger agreement at smaller significance levels alpha (Section 5).

  • Decisive quantitative result: VERIFIED on all three registered families. The 5,000-path Beta and Bernoulli cells reach final KS 0.012 and 0.023. A 3,000-path bootstrap over 44 pinned primary DSSAT HWAM yields gives standardized mean -0.093, variance 0.975, and Gaussian KS 0.090.
  • Native scale: The computation uses the paper's literal KL_inf dual and thresholds, its Beta(3,2)/Bernoulli settings, 5,000 paths per synthetic cell, and 3,000 bootstrap paths over pinned primary DSSAT maize-yield observations.
  • Source locator: source/paper/icml_final_submission.tex Experiments 1-3; source/dssat-maize official primary observations; reproduce.py claim 5
  • Upstream pin:
    • digest: sha256:27ca92473d1ca2e37c227850717f771cbfd820e7db0ab3459a868ecb7bcea220
    • dssat_commit: a4f95d3ef36f1358bdeb5db49d498d5db373ba7a
    • version: arXiv:2606.04520 accepted source
  • Independent evidence:
    • outputs/claim5.json
    • replay_a/claim5.json
    • replay_b/claim5.json
  • Executed outputs:
    • outputs/claim5.json
    • outputs/results.json
  • Independent oracle paths:
    • replay_a/claim5.json
    • replay_b/claim5.json
  • Control paths:
    • outputs/claim5.json
    • outputs/results.json
  • Destructive or boundary control: The synthetic distributions, two sample sizes, two alpha levels, two boundary forms, and real nonparametric pool are mutually destructive boundary controls against a distribution-specific or source-only result.
  • Rate relation: claim-consistent; mode empirical_scaling; measured slope -0.617.
    • Horizons: 100, 500, 1000, 2000
    • Repetitions per horizon: 5000
    • Measurement: Synthetic KS improves with n/smaller alpha and the 3,000-path DSSAT cell achieves KS 0.090239
    • Rate artifact: outputs/claim5.json
  • Limitation: The verdict is limited to the literal statistic, distributions, thresholds, and primary-data realization executed here; no neighboring theorem, proxy statistic, or source narration is counted.
  • Scope boundary: The verdict resolves the registered statement at the accepted paper's stated synthetic scale and through a pinned official DSSAT primary-data realization.