BatterySwapAI2026-Example / REPRODUCIBILITY.md
Thibaut Forest
Claude Fable 5
docs: REPRODUCIBILITY - add 354fd2d public read (1495.92, not selected)
e21fe90
|
Raw History Blame Contribute Delete
4.91 kB

Reproducibility

This document identifies the submitted version(s) and the exact pipeline that reproduces every trained or stored artifact in this repository, per the competition rules.

Submitted commits

The final submission is selected in the competition portal from publicly scored versions of this repository. Publicly scored versions relevant to the selection:

commit public score configuration (env defaults at that commit)
62e9d91 1480.85 primary selection — order .50 logbag + .30 Wiener + .20 gapHGB, alpha 1.4, planner split_days=3 / timing_weight=2.0
354fd2d 1495.92 not selected — order re-weight .50/.20/.30 (refuted by the public score; reverted at head)
4c93883 1575.31 fallback — same ordering, alpha 1.0, planner split_days=7 / timing_weight=1.0
71bfff2 1583.11 not selected — alpha 1.0, planner split_days=3 / timing_weight=2.0 (equivalent to 60d65da)
e4745d2 1594.89 not selected — warmup gate ON (refuted by the public score)

The exact submitted commit is the one recorded in the competition portal at selection time; unless stated otherwise there, it is 62e9d91.

Pipeline: what is trained, and when

The scored entry point is script.py, which calls batteryswap_example/submit.py:main(). No pre-trained model is used on the primary path. All estimators (bagged logistic RUL model, Wiener first-hitting-time model, gap-feature HistGradientBoosting order source) are fit at submission run time, inside the evaluation container, from the dataset provided to the run. The submitted model is therefore fully specified by the code at the submitted commit plus the configuration below — there is no training step to reproduce separately for the primary path.

Determinism: every estimator is seeded (random_state=0 throughout submit.py/models_cv.py; ancillary sampling uses numpy.random.RandomState(0)). Given the same dataset and the pinned competition environment (/app/env in the huggingface/competitions image), repeated runs emit an identical submission.csv. Defensive fallbacks exist (on failure or budget overrun a split degrades to a simpler static fit, with a log line); all publicly scored runs of the commits above completed on the primary path (leaderboard components invert to integer swap/day counts consistent with it).

Configuration and seeds

All dials are environment variables read in batteryswap_example/submit.py with in-repo defaults. The Dockerfile sets only BATTERYSWAP_SPLITS and BATTERYSWAP_SUBMISSION_PATH; the env defaults in the code at the submitted commit are the submitted configuration — no external configuration files or overrides are used. Random seeds are fixed in code (seed 0 everywhere); no seed is supplied at run time.

Exact reproduction commands

git checkout <submitted-commit>   # e.g. 4c93883
docker build -t batteryswapai-2026-example .
docker run --name batteryswapai -v ./dataset:/tmp/data \
  batteryswapai-2026-example \
  bash -c "/app/env/bin/python3 script.py && /app/env/bin/python3 -m batteryswap_public.metric"
docker cp batteryswapai:/app/submission.csv ./submission.csv

dataset/ must contain the official competition dataset (not redistributed here; see THIRD_PARTY_NOTICES.md). BATTERYSWAP_SPLITS defaults to train; the scoring system sets its own splits.

Stored artifact: batteryswap_example/planners/best.pickle

  • sha256: 1d263e1282560dc608868cb9ce3360007c2230b360bd4ea272b260bd3b08b0d0
  • Role: fallback only. It is loaded exclusively if the run-time-fitted path fails for a split; it does not contribute to the primary scored output.
  • Provenance: produced by python batteryswap_example/train.py (which fits the legacy OrderedPlanner RUL model, seeded, and writes this path) run inside the Docker environment above, at commit 4fb1dc8. Re-running that command at that commit in that environment reproduces an equivalent artifact; equivalence should be verified on emitted plans rather than pickle byte-identity (pickle serialization is not guaranteed byte-stable across library builds). Note: running train.py overwrites this file.

Verification

  1. Run the Docker commands above at the submitted commit; the container prints the official metric for the chosen split via batteryswap_public.metric.
  2. Cross-check against the public leaderboard scores in the table above for the corresponding commits.
  3. The internal cross-validation harness reproduces the model-selection number at the repository head: from the repo root with the dataset in place, python -m batteryswap_example.cv cand ft2a10 → 1541.3 (building-grouped 5-fold out-of-fold cost, mean over 48 scenarios).

Full experiment provenance for every modeling decision is recorded in EXPERIMENTS.md (phases 1–69) and agent_reports_bgkf/.