Commit History

trainer: the post-stop assertions are rank-0 facts (E-044: hub_cb.steps is empty on rank 1 by construction, so the guard killed every correctly-checkpointed segment)
e26f3f1
verified

Cion-lab commited on

trainer: per-rank marks through the post-training sequence (v6's rank-1 death was unlocalisable because every line there is rank-0-only)
296186e
verified

Cion-lab commited on

trainer: an unverified checkpoint never rolls latest.json and never prunes (E-042)
a19fe02
verified

Cion-lab commited on

hubckpt: Content-Range + size guards on the bounded read-back, and prune_verified() (review E-042)
2776a36
verified

Cion-lab commited on

hubckpt: bounded Range read-back in verify_checkpoint (probe v5 measured a 797s full-object download; the run cannot afford eleven of those)
5282366
verified

Cion-lab commited on

train: a segment that stopped short of the horizon skips the validation pass instead of entering a collective the two ranks can reach a batch apart (probe 1 v1/v4 found rank 1 dying there with no traceback, while every horizon leg passed the same call)
b6a124b
verified

Cion-lab commited on

review round 3 (probe/model-card/eval): rank-visible eval failure + cross-rank peak memory + complete run_summary; card derived-and-asserted from config; harness install --no-deps and task ids confirmed; torchrun redirects 3 + tee 3
52c7981
verified

Cion-lab commited on

train: escape the literal percent signs in two --help strings (argparse interpolates help text, so '+52 %' crashed --help with 'must be real number, not dict' -- caught by the CPU rehearsal's stage 3)
76fbbb7
verified

Cion-lab commited on

perm_sha: identify the visit order by hashing the permutation, not just its seed (E-035 residual) -- cursor gains shuffle_perm_sha, asserted on resume; session repins the two files
ef0c95d
verified

Cion-lab commited on

perm_sha: identify the visit order by hashing the permutation, not just its seed (E-035 residual) -- cursor gains shuffle_perm_sha, asserted on resume; session repins the two files
398d16e
verified

Cion-lab commited on

review round: pointer written+read-back before prune, verify retried, latest rolled to final, stale-pointer scan on resume, ddp_timeout 4h, chunked batched commits, manifest self-consistency gate, full content verification after upload
492bf86
verified

Cion-lab commited on

review round: pointer written+read-back before prune, verify retried, latest rolled to final, stale-pointer scan on resume, ddp_timeout 4h, chunked batched commits, manifest self-consistency gate, full content verification after upload
e11175e
verified

Cion-lab commited on

mix_fingerprint: fold the manifest's per-shard sha256 into the corpus identity (names+sizes could collide across rebuilds); phase4_session repins shard_dataset
c588f4b
verified

Cion-lab commited on

pin: train/train_ounce100m.py (RepoMissing resume branch, disk gate, PHASE4_PREP_ONLY)
7dc409a
verified

Cion-lab commited on

pin: train/hubckpt.py (RepoMissing resume branch, disk gate, PHASE4_PREP_ONLY)
cc004dc
verified

Cion-lab commited on

Upload train/hubckpt.py with huggingface_hub
001ecbf
verified

Cion-lab commited on

Upload train/train_ounce100m.py with huggingface_hub
212e0b7
verified

Cion-lab commited on

Upload train/preflight.py with huggingface_hub
f256711
verified

Cion-lab commited on

Upload train/train_ounce100m.py with huggingface_hub
6ed0dbb
verified

Cion-lab commited on

Upload train/hubckpt.py with huggingface_hub
d5df04b
verified

Cion-lab commited on

Upload train/train_ounce100m.py with huggingface_hub
e11176d
verified

Cion-lab commited on

Upload train/train_ounce100m.py with huggingface_hub
18d95d5
verified

Cion-lab commited on

Upload train/preflight.py with huggingface_hub
c5f35bf
verified

Cion-lab commited on

Upload train/train_ounce100m.py with huggingface_hub
c3592b7
verified

Cion-lab commited on

Upload train/preflight.py with huggingface_hub
758d461
verified

Cion-lab commited on

train: --grad-ckpt, and record attn/grad_ckpt in every result line
a477d8d
verified

Cion-lab commited on

train: --benchmark N runs the same step with no Trainer, to attribute the P2 throughput gap
182a8d9
verified

Cion-lab commited on

train: explicit attention kernel (default eager, per p0c on Turing); report requested vs effective
3880d41
verified

Cion-lab commited on

train: preflight -- static-review fixes: absolute-step cursor, hub_listing pagination, cursor required on resume, repo_type everywhere, rank-0 repo create
658b0a8
verified

Cion-lab commited on

train: train_ounce100m -- static-review fixes: absolute-step cursor, hub_listing pagination, cursor required on resume, repo_type everywhere, rank-0 repo create
11456a3
verified

Cion-lab commited on

train: hubckpt -- static-review fixes: absolute-step cursor, hub_listing pagination, cursor required on resume, repo_type everywhere, rank-0 repo create
7c925b7
verified

Cion-lab commited on

train: shard_dataset -- static-review fixes: absolute-step cursor, hub_listing pagination, cursor required on resume, repo_type everywhere, rank-0 repo create
cfc38f1
verified

Cion-lab commited on

preflight: P2 fast-fails after the first dead cell (each repeat bills a GPU session)
51eb041
verified

Cion-lab commited on

preflight: P2 parses RUN_JSON per cell and requires the frozen 22L/2048 cell to report
7aab8c2
verified

Cion-lab commited on

train: frozen-width keeps GQA 3:1; report peak_gpu_gb and per-run tok/s
db65aa1
verified

Cion-lab commited on

shard_dataset: drop the removed drop_last param's leftover body check (NameError at first call)
d5f46da
verified

Cion-lab commited on

preflight: assert the trapezoid SHAPE (off-by-one at the last step is expected, not a bug)
e687c2c
verified

Cion-lab commited on

train: train_ounce100m -- readback counts as verification; GQA kv must divide heads; pargs reports before optional probe
9bf38c0
verified

Cion-lab commited on

train: preflight -- readback counts as verification; GQA kv must divide heads; pargs reports before optional probe
48c0e43
verified

Cion-lab commited on

train: hubckpt -- readback counts as verification; GQA kv must divide heads; pargs reports before optional probe
51574f8
verified

Cion-lab commited on

train: train_ounce100m -- download_checkpoint prefix/landing-dir fix (found by P1 v2)
4a89fe7
verified

Cion-lab commited on

train: preflight -- download_checkpoint prefix/landing-dir fix (found by P1 v2)
9a7dcc5
verified

Cion-lab commited on

train: hubckpt -- download_checkpoint prefix/landing-dir fix (found by P1 v2)
5f6c3e9
verified

Cion-lab commited on

train: fix SwiGLU width in build_config variants (was 2x)
b09a370
verified

Cion-lab commited on

preflight: stage pargs -- construct TrainingArguments/model/schedule on the Kaggle image, cross-check 22L=106,194,240 and 20L=99,114,048
87df298
verified

Cion-lab commited on

train: train_ounce100m -- hubckpt.ensure_repo writes LFS attributes before the first checkpoint push (found by P1)
7a667d0
verified

Cion-lab commited on

train: hubckpt -- hubckpt.ensure_repo writes LFS attributes before the first checkpoint push (found by P1)
f7ea5a1
verified

Cion-lab commited on

train: preflight -- hubckpt.ensure_repo writes LFS attributes before the first checkpoint push (found by P1)
1279da6
verified

Cion-lab commited on

train: state which tokenizer files shipped with the export
38d1f55
verified

Cion-lab commited on

train: create the ckpt repo if absent before the first push
0538d57
verified

Cion-lab commited on