base README.md

hf/base/README.md

@@ -0,0 +1,78 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- daedalus
+- cpu-inference
+- gguf
+- q4_0
+---
+
+# daedalus-150m
+
+A 160.5M-parameter causal LM built for the best quality-per-token-per-second on **CPU** inference, exported to GGUF Q4_0 for llama.cpp.
+
+## What this model is trying to beat
+
+> Beat Pythia-160M, OPT-125M and GPT-neo-125M on quality; target MobileLLM-125M as a stretch; concede SmolLM2-135M on quality while beating it decisively on CPU decode.
+
+This bar was fixed before any result landed. Numbers below are reported against it whether or not they clear it.
+
+## Architecture
+
+| | |
+|---|---|
+| exported as | `Lfm2ForCausalLM` |
+| parameters | 160,488,960 (122,740,224 non-embedding) |
+| blocks | 18 (`ccccAccAcAcAcAccAc` -- `c` = gated short conv, `A` = GQA attention) |
+| hidden size | 768 |
+| SwiGLU inner dim | 2048 |
+| heads | 12 query / 4 KV, head_dim 64, QK-norm |
+| RoPE theta | 1,000,000 |
+| context | 2048 |
+| tied embeddings | True |
+| tokenizer | [`HuggingFaceTB/SmolLM2-135M`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M), reused byte-identical, vocab 49,152 |
+
+## Training
+
+- run: `hero`
+- tokens seen: 59,900,334,080 (373 tokens/parameter)
+- Muon lr 0.02 on 2D hidden matrices; AdamW lr 0.0003 on embeddings/head/norms
+- WSD schedule, linear decay to zero over the final 45% of the run
+
+## Evaluation
+
+_Not yet measured for this export._
+
+## Q4_0 quantization
+
+_Not yet measured for this export._
+
+## Checkpoints and how to continue training
+
+Checkpoints are pushed to the private Hub model repo **`Unseen1980/daedalus-checkpoints`**: weights-only bf16 rolling copies every ~2 h under `rolling/<run>/weights.pt`, plus a milestone with full Muon + AdamW optimizer state at the WSD decay-start step on its own revision.
+
+The stable-phase branch point for this model is revision **`hero-stable-end-step68461`** (step 68,461, 30,532,341,760 tokens seen, lr multiplier 1.0). To continue stable-phase training from it on more or different data and then re-decay:
+
+```bash
+python train.py --run-name hero-ext --config daedalus-150m \
+  --data-dir <YOUR_SHARD_DIR> \
+  --total-tokens <NEW_BUDGET_GREATER_THAN_30532341760> \
+  --resume 'hub://Unseen1980/daedalus-checkpoints/milestone/hero/checkpoint.pt?rev=hero-stable-end-step68461'
+```
+
+Fill both placeholders. `--total-tokens` **must exceed the 30,532,341,760 tokens already seen** — a smaller budget makes the run stop at the top of its first iteration, printing a `resumed from ...` line and exiting 0 having trained nothing. And `--data-dir` is not optional: without it training falls back to randomly generated tokens, which silently destroys the checkpoint you branched from.
+
+Branching from the pre-decay checkpoint is the point of WSD: resuming an already-annealed model needs an lr re-warmup from a converged state, which is measurably worse.
+
+## Deviations from the blueprint
+
+Each was costed and approved rather than silently dropped; see `DAEDALUS-BLUEPRINT-v6.md` and issue #4.
+
+- **No distillation** from SmolLM2-1.7B during decay. 288 GB of top-16 logits does not fit the disk and the online-teacher variant cost ~$29 of a $94.66 budget; its own evidence was only "+1-3 points plausible".
+- **Corpus stops at ~14.2B tokens, not 45B.** Training repeats a balanced corpus rather than seeing 45B unique tokens; at this scale repetition up to ~4 epochs costs little against fresh tokens, and mixture balance mattered more than raw size.
+- **Document-aligned packing not implemented** -- sequences may cross document boundaries.
+- **NoPE skipped** -- it breaks GGUF export.
+- **Single seed** for the hero run, so no seed-sigma is reported.
+- **`everyday-conversations` contributes ~0.00%** of pretraining instead of its 2% share (the whole dataset is 0.4M tokens, which the 4-epoch cap reduces to nothing); dialogue enters at the `post` SFT stage instead.