model card: layout, branch point, and how to extend
Browse files
README.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: pytorch
|
| 3 |
+
tags: [daedalus, checkpoint, wsd]
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# Daedalus — training checkpoints
|
| 7 |
+
|
| 8 |
+
Working checkpoints for [unseen1980/daedalus](https://github.com/unseen1980/daedalus):
|
| 9 |
+
a ~150M-parameter LFM2-class conv/attention hybrid, trained on a rented GPU box
|
| 10 |
+
that does not survive a recycle. AGENT.md §0.2: *"Never store state only on this
|
| 11 |
+
box... If it isn't pushed, it doesn't exist."*
|
| 12 |
+
|
| 13 |
+
These are `torch.save` payloads, not `transformers` weights. Final GGUF exports
|
| 14 |
+
live elsewhere.
|
| 15 |
+
|
| 16 |
+
## Layout
|
| 17 |
+
|
| 18 |
+
| branch | path | contents | size |
|
| 19 |
+
|---|---|---|---|
|
| 20 |
+
| `rolling` | `rolling/<run>/weights.pt` | weights only, bf16 | 321 MB |
|
| 21 |
+
| `<run>-stable-end-step<N>` | `milestone/<run>/checkpoint.pt` | weights + Muon + AdamW state, fp32 | 1.4 GB |
|
| 22 |
+
|
| 23 |
+
`rolling` is overwritten roughly every 2 hours and exists purely so that losing
|
| 24 |
+
the instance costs at most two hours rather than the whole run. Each milestone
|
| 25 |
+
branch is written once and is never overwritten.
|
| 26 |
+
|
| 27 |
+
`latest-rolling-<run>.json` and `latest-milestone-<run>.json` on `main` record
|
| 28 |
+
step and tokens-seen, so progress can be read without downloading a checkpoint.
|
| 29 |
+
|
| 30 |
+
## The milestone is a branch point
|
| 31 |
+
|
| 32 |
+
It is written at the step the WSD schedule leaves its stable phase
|
| 33 |
+
(`decay_frac=0.45`, i.e. 55% of the run), with full optimizer state. That is the
|
| 34 |
+
practical advantage of WSD over cosine: pre-decay weights can be trained further
|
| 35 |
+
on more or different data and then re-decayed, whereas a model already annealed
|
| 36 |
+
to lr≈0 needs an lr re-warmup from a converged state and does measurably worse.
|
| 37 |
+
Optimizer state is included because a branch that has to rebuild Muon's momentum
|
| 38 |
+
buffers and AdamW's moments loses ground on restart.
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
python train.py --run-name hero-ext --config daedalus-150m \
|
| 42 |
+
--data-dir data/shards --total-tokens <new budget> \
|
| 43 |
+
--resume 'hub://Unseen1980/daedalus-checkpoints/milestone/hero/checkpoint.pt?rev=hero-stable-end-step<N>'
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
`--resume` accepts `hub://owner/repo/path?rev=branch` directly and resolves it to
|
| 47 |
+
a local file, so restoring from here takes the same code path as a local resume.
|
| 48 |
+
|
| 49 |
+
## Verification
|
| 50 |
+
|
| 51 |
+
The upload and restore path was checked live against this repo before any long
|
| 52 |
+
run depended on it — LFS, branch creation, and a clean-directory download and
|
| 53 |
+
resume. Worst relative weight delta through the bf16 round trip: 0.0039.
|
| 54 |
+
See `runs/preflight/hub-restore.md` in the GitHub repo.
|