ounce100m-code / train /train_ounce100m.py

Commit History

29 days ago

  1. trainer: the post-stop assertions are rank-0 facts (E-044: hub_cb.steps is empty on rank 1 by construction, so the guard killed every correctly-checkpointed segment)

    Verified signature
    e26f3f1
  2. trainer: per-rank marks through the post-training sequence (v6's rank-1 death was unlocalisable because every line there is rank-0-only)

    Verified signature
    296186e
  3. trainer: an unverified checkpoint never rolls latest.json and never prunes (E-042)

    Verified signature
    a19fe02
  4. train: a segment that stopped short of the horizon skips the validation pass instead of entering a collective the two ranks can reach a batch apart (probe 1 v1/v4 found rank 1 dying there with no traceback, while every horizon leg passed the same call)

    Verified signature
    b6a124b
  5. review round 3 (probe/model-card/eval): rank-visible eval failure + cross-rank peak memory + complete run_summary; card derived-and-asserted from config; harness install --no-deps and task ids confirmed; torchrun redirects 3 + tee 3

    Verified signature
    52c7981
  6. train: escape the literal percent signs in two --help strings (argparse interpolates help text, so '+52 %' crashed --help with 'must be real number, not dict' -- caught by the CPU rehearsal's stage 3)

    Verified signature
    76fbbb7
  7. perm_sha: identify the visit order by hashing the permutation, not just its seed (E-035 residual) -- cursor gains shuffle_perm_sha, asserted on resume; session repins the two files

    Verified signature
    ef0c95d
  8. review round: pointer written+read-back before prune, verify retried, latest rolled to final, stale-pointer scan on resume, ddp_timeout 4h, chunked batched commits, manifest self-consistency gate, full content verification after upload

    Verified signature
    492bf86

30 days ago

  1. pin: train/train_ounce100m.py (RepoMissing resume branch, disk gate, PHASE4_PREP_ONLY)

    Verified signature
    7dc409a
  2. Upload train/train_ounce100m.py with huggingface_hub

    Verified signature
    212e0b7
  3. Upload train/train_ounce100m.py with huggingface_hub

    Verified signature
    6ed0dbb
  4. Upload train/train_ounce100m.py with huggingface_hub

    Verified signature
    e11176d
  5. Upload train/train_ounce100m.py with huggingface_hub

    Verified signature
    18d95d5
  6. Upload train/train_ounce100m.py with huggingface_hub

    Verified signature
    c3592b7
  7. train: --grad-ckpt, and record attn/grad_ckpt in every result line

    Verified signature
    a477d8d
  8. train: --benchmark N runs the same step with no Trainer, to attribute the P2 throughput gap

    Verified signature
    182a8d9
  9. train: explicit attention kernel (default eager, per p0c on Turing); report requested vs effective

    Verified signature
    3880d41
  10. train: train_ounce100m -- static-review fixes: absolute-step cursor, hub_listing pagination, cursor required on resume, repo_type everywhere, rank-0 repo create

    Verified signature
    11456a3
  11. train: frozen-width keeps GQA 3:1; report peak_gpu_gb and per-run tok/s

    Verified signature
    db65aa1
  12. train: train_ounce100m -- readback counts as verification; GQA kv must divide heads; pargs reports before optional probe

    Verified signature
    9bf38c0
  13. train: train_ounce100m -- download_checkpoint prefix/landing-dir fix (found by P1 v2)

    Verified signature
    4a89fe7
  14. train: fix SwiGLU width in build_config variants (was 2x)

    Verified signature
    b09a370
  15. train: train_ounce100m -- hubckpt.ensure_repo writes LFS attributes before the first checkpoint push (found by P1)

    Verified signature
    7a667d0
  16. train: state which tokenizer files shipped with the export

    Verified signature
    38d1f55
  17. train: create the ckpt repo if absent before the first push

    Verified signature
    0538d57
  18. train: add train_ounce100m (Phase 3/4 glue)

    Verified signature
    6ddf1d2