JacobLinCool/chebyshev-job-artifacts/release/v1 - Storage Bucket

Chebyshev Policies independent reproduction

Executive summary

This release audits all five challenge claims for ICML 2026 paper 12142 against the current arXiv v4 manuscript. The deterministic Mountain Car return panel is reproduced, while the theorem experiment supports only the paper's continuous, unconstrained mechanism and cannot establish the historical-priority headline or a bounded discrete-control global optimum. CH-3-ARS matches the v4 performance claim, but the published parameter-count arithmetic is internally inconsistent. Claim 4 is not reproduced: complete PPO and independently seeded REINFORCE grids both expose localized start-state failures and lose to their named baselines. On Pendulum, the released ARS point estimate and the +67.8248 CH-6 ordering margin reproduce, although fresh-trained CH-6 misses the strict point-estimate tolerance by 0.3018. The physical Aero 2 subclaim remains inconclusive because the required hardware was unavailable.

Claim-by-claim verdicts

Claim	Independent result	Verdict
1. Closed-form Mountain Car control	The 10,000-perturbation certificate has minimum loss gap 0.008298, maximum equality residual 5.85e-18, and maximum KKT-ratio deviation 1.78e-15. It supports the Cauchy–Schwarz velocity-proportional mechanism only under the continuous, unconstrained assumptions. The executable controller also contains two constants and a position bootstrap. A bounded primary-source search did not verify the first / 36-year-old open problem language.	Partially supported, materially narrower than the headline.
2. Analytical and neural returns	Across the complete 100-start grid, the analytical policy gives mean 99.3941 and range 99.1494–99.5179; ARS, PPO, and SAC give 96.6663, 93.9056, and 94.6052. Recomputed regrets are 2.7279, 5.4885, and 4.7889.	Reproduced.
3. CH-3-ARS	The preregistered 20-seed search selects seed 1107, whose 100-start mean is 98.9503 and regret is 0.4438. The ratio from exact reproduced regrets is 6.146×; the paper's 6.18× comes from dividing the separately rounded values 2.72/0.44. The Chebyshev policy has 16 coefficients, but a biased 2→64→64→1 MLP has 4,417—not 4,355—parameters, so the exact reduction is 276.0625×, not 277×. The released ARS comparator has only 65 parameters.	Performance reproduced; ratio consistent only at headline precision; parameter arithmetic corrected; selection is best-of-20 on the reporting grid.
4. CH-3-PPO and CH-3-REINFORCE	The complete 20-seed PPO protocol selects seed 738, but its fixed-grid mean is only 84.5836, regret is 14.8106, and its exact improvement ratio over PPO is 0.3706×; 14 sampled starts concentrated in a narrow band return below zero. A fresh fixed-seed REINFORCE replication selects seed 246, whose fixed-grid mean is 96.0097, regret is 3.3844, and improvement ratio over ARS is 0.8060×; two starts return below zero. Both baselines therefore have lower regret.	Not reproduced. PPO strongly contradicts the reported advantage under the released protocol. REINFORCE is an inconsistent independent replication rather than an exact replay because the paper released neither its selected seed nor its coefficients.
5. Pendulum and Aero 2	The best of 30 fresh CH-6-ARS seeds is seed 3075, with mean -150.4982; it performs slightly better than the paper's -150.80 but differs by 0.3018, outside the preregistered ±0.05 point-estimate tolerance. Released ARS gives -218.3231, reproducing -218.30; the CH-6 advantage is +67.8248. Correcting the first-observation reset changes ARS by +2.2210. The physical Aero 2 result could not be tested without the Quanser rig, matching sensors, calibration, and real-time interface; simulation was not accepted as a substitute for physical sim-to-real evidence.	Partially supported. Pendulum's released baseline and Chebyshev-over-ARS ordering reproduce; the fresh CH-6 point estimate misses the strict gate, and Aero 2 remains inconclusive.

The challenge text preserves some arXiv v1 values. In particular, its CH-3-ARS regret 0.65 and 4.18× improvement were superseded in arXiv v4 by 0.44 and 6.18×; both versions are labeled in the canonical claim table rather than silently mixed.

How to reproduce

Use Python 3.10 and the frozen uv.lock. The runner verifies both author repositories at their immutable commits before importing them. Each accepted raw batch is append-only and must close with a self-consistent manifest.

uv sync --frozen

uv run --python 3.10 scripts/run_reproduction.py \
  --config configs/claim23_full.json \
  --batch-id YOUR-UNIQUE-BATCH \
  --output-root runs/raw \
  --lock-path upstream.lock.json \
  --spec-path experiments/icml2026_chebyshev/SPEC.md

Claims 4 and 5 use configs/claim4_full.json and configs/claim5_pendulum_full.json. The exact public, SHA-256-verified rerun recipe for every attempt is recorded in the Jobs inventory. After obtaining the three complete raw batches, regenerate all derived evidence in the declared order:

uv run --python 3.10 scripts/build_tables.py \
  --raw-root runs/raw/hf-claim23-full-v4-20260715T1844Z \
  --raw-root runs/raw/hf-claim4-full-v4-20260715T1845Z \
  --raw-root runs/raw/hf-claim5-pendulum-full-v4-20260715T1846Z \
  --output-root runs/tables

uv run --python 3.10 scripts/make_figures.py \
  --tables-root runs/tables --output-root runs/reports

uv run --python 3.10 scripts/analyze_claims.py \
  --tables-root runs/tables --output-root runs/analysis

Each derived manifest records the input hashes, builder-script hash, and exact absolute command. Tests exercise raw-batch admission, cross-batch identity, provenance, figure inventory, poster gates, secret scanning, and release scope.

Evidence and provenance

• OpenReview paper
• arXiv v4 used for quantitative targets
• Public raw-evidence Bucket
• Claims 1–3 Job
• Claim 4 Job
• Claim 5 Pendulum Job
• Nine-attempt Jobs inventory
• Release metadata and raw-record hashes

The scheduler_command field is the inspected scheduler argv, while reproduction_command is a separate hash-verified rerun recipe. The curated bundle intentionally excludes unlicensed author source, author model weights, and duplicate raw records. It includes the exact raw manifests and input snapshots, newly trained JSON coefficients, canonical tables, analyses, interactive reports, poster, tests, and a full SHA-256 inventory. The raw records.jsonl files and their artifacts remain public at the Bucket URLs and are anchored by hashes in both RELEASE.json and jobs/HF_JOBS.json.

Interactive reports

• Mountain Car return versus initial position
• All-seed robustness and selected policies
• Parameter count versus regret
• Pendulum paired reset sensitivity
• One-page reproduction poster

Every interactive report embeds its plotted data and is paired with a canonical CSV, so readers can inspect exact values without a notebook or hidden server state. The poster links back to the public Trackio logbook for the full claim trail.

Limitations

The Claim 1 experiment is a numerical certificate for the theorem's specified continuous equality-constrained problem, not a proof of global optimality for Gym's bounded discrete action task. A bounded literature audit cannot prove that no earlier solution exists. Claims 3 and 4 follow the paper's best-seed selection protocol, which uses the reporting grid and therefore estimates selected performance rather than an untouched test-set expectation. Pendulum results show measurable reset sensitivity, so both the author-exact and corrected reset protocols are retained. Aero 2 remains inconclusive, and no simulation proxy is presented as physical transfer evidence. Peak process RSS is reported alongside the 30 GB Job flavor; GPU allocation does not imply GPU utilization because these workloads are principally CPU-bound.