model card: public release — results, speed claim, architecture, limitations

README.md

@@ -1,54 +1,146 @@
 ---
-library_name: pytorch
-tags: [daedalus, checkpoint, wsd]
+license: apache-2.0
+library_name: gguf
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - daedalus
+  - cpu-inference
+  - gguf
+  - hybrid
+  - conv-attention
+base_model: Unseen1980/daedalus-checkpoints
 ---
 
-# Daedalus — training checkpoints
+# Daedalus-150M
 
-Working checkpoints for [unseen1980/daedalus](https://github.com/unseen1980/daedalus):
-a ~150M-parameter LFM2-class conv/attention hybrid, trained on a rented GPU box
-that does not survive a recycle. AGENT.md §0.2: *"Never store state only on this
-box... If it isn't pushed, it doesn't exist."*
+A 150M-parameter language model built for **CPU inference**. Two thirds of its
+layers are short convolutions with a fixed-size state instead of attention, so
+decoding does not slow down as the context grows.
 
-These are `torch.save` payloads, not `transformers` weights. Final GGUF exports
-live elsewhere.
+Trained from scratch on 59.9B tokens. Code and paper:
+[unseen1980/daedalus](https://github.com/unseen1980/daedalus).
 
-## Layout
+## Quick start
 
-| branch | path | contents | size |
+```bash
+brew install llama.cpp   # or build from ggml-org/llama.cpp
+hf download Unseen1980/daedalus-checkpoints instruct/model-q4_0.gguf --local-dir ./daedalus
+
+llama-cli -m ./daedalus/instruct/model-q4_0.gguf -cnv \
+  --temp 0.8 --top-p 0.9 --repeat-penalty 1.15
+```
+
+**Pass sampling flags.** llama.cpp defaults `--repeat-penalty` to 1.0, i.e. off,
+and this model will loop on a repeated token without it.
+
+## Files
+
+| File | Size | What |
+|---|---|---|
+| `instruct/model-q4_0.gguf` | 102 MB | chat model, 4-bit — **start here** |
+| `gguf/hero-base-q4_0.gguf` | 102 MB | base model, text completion |
+| `gguf/instruct-f16.gguf` | 323 MB | instruct, f16 — for re-quantising |
+| `gguf/hero-base-f16.gguf` | 323 MB | base, f16 |
+| `hf/instruct/`, `hf/base/` | 321 MB | HF-format safetensors + tokenizer |
+| `final/hero/checkpoint.pt` | 1.4 GB | base weights + optimizer state |
+| `final/post-sft/final.pt` | 642 MB | instruct weights, full precision |
+
+The **base** model deliberately carries no chat template. Giving one to a base
+model makes llama.cpp wrap prompts in markup it never saw during training, which
+produces fluent but unrelated output. Use plain prompts, or `llama-completion`.
+
+## Results
+
+Five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande, with
+every peer re-scored on the same harness rather than quoted from its paper.
+
+| Model | Training tokens | 5-task mean |
+|---|---|---|
+| **Daedalus-150M** | **59.9B** | **47.31** |
+| MobileLLM-125M | 1T | 46.3 *(published)* |
+| GPT-2 124M | — | 42.2 |
+| OPT-125M | 180B | 42.1 |
+| GPT-neo-125M | 300B | 41.9 |
+| Pythia-160M | 300B | 41.0 |
+| SmolLM2-135M | 2T | 51.2 |
+
+Validation bits-per-byte **0.8685** over 645M held-out tokens.
+
+SmolLM2-135M stays ahead on quality — conceded in advance. The trade this model
+makes is speed.
+
+## Speed
+
+CPU decode, 4-bit, 8 threads, against a parameter-matched all-attention twin
+trained on identical data:
+
+| Context | Daedalus | Dense twin | Ratio |
 |---|---|---|---|
-| `rolling` | `rolling/<run>/weights.pt` | weights only, bf16 | 321 MB |
-| `<run>-stable-end-step<N>` | `milestone/<run>/checkpoint.pt` | weights + Muon + AdamW state, fp32 | 1.4 GB |
+| 0 | 1112 tok/s | 923 tok/s | 1.20× |
+| 512 | 960 tok/s | 664 tok/s | 1.45× |
+| **2048** | **739 tok/s** | **420 tok/s** | **1.76×** |
+
+**The trend is the result.** At an empty context the hybrid has nothing to gain —
+its advantage *is* the key–value cache it does not keep. Against an external
+135M peer the same pattern reaches **2.08×** at 2048 tokens.
 
-`rolling` is overwritten roughly every 2 hours and exists purely so that losing
-the instance costs at most two hours rather than the whole run. Each milestone
-branch is written once and is never overwritten.
+Per token of context this model reads 6,144 bytes of cache against a 24-layer
+all-attention model's 12,288 — half. At 2048 tokens that is 12.6 MB re-read per
+generated token instead of 25.2 MB.
 
-`latest-rolling-<run>.json` and `latest-milestone-<run>.json` on `main` record
-step and tokens-seen, so progress can be read without downloading a checkpoint.
+## Architecture
 
-## The milestone is a branch point
+```
+18 blocks, d_model 768, vocab 49,152, context 2048
 
-It is written at the step the WSD schedule leaves its stable phase
-(`decay_frac=0.45`, i.e. 55% of the run), with full optimizer state. That is the
-practical advantage of WSD over cosine: pre-decay weights can be trained further
-on more or different data and then re-decayed, whereas a model already annealed
-to lr≈0 needs an lr re-warmup from a converged state and does measurably worse.
-Optimizer state is included because a branch that has to rebuild Muon's momentum
-buffers and AdamW's moments loses ground on restart.
+block:  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18
+type:   C  C  C  C  A  C  C  A  C  A  C  A  C  A  C  C  A  C
 
-```bash
-python train.py --run-name hero-ext --config daedalus-150m \
-  --data-dir data/shards --total-tokens <new budget> \
-  --resume 'hub://Unseen1980/daedalus-checkpoints/milestone/hero/checkpoint.pt?rev=hero-stable-end-step<N>'
+A = full attention (6)      GQA, 12 query heads / 4 KV heads
+C = short convolution (12)  depthwise, kernel 3, fixed 2-step state
 ```
 
-`--resume` accepts `hub://owner/repo/path?rev=branch` directly and resolves it to
-a local file, so restoring from here takes the same code path as a local resume.
+Tied embeddings, 2048 FFN, RoPE θ=1e6. `Q4_0` chosen for ARM kernel speed rather
+than its error curve.
+
+## Training
 
-## Verification
+59.9B tokens over a 16.9B-token corpus (~3.5 epochs, capped at 4 per source) of
+public English data weighted toward educational text: FineWeb-Edu 37.5%,
+DCLM-baseline 22.5%, Stack-Edu 9%, FinePDFs-Edu 8%, FinePhrase 7%,
+Cosmopedia-v2 5%, FineMath + InfiWebMath 6%, FineWiki-en 3%, dialogue 2%.
 
-The upload and restore path was checked live against this repo before any long
-run depended on it — LFS, branch creation, and a clean-directory download and
-resume. Worst relative weight delta through the bf16 round trip: 0.0039.
-See `runs/preflight/hub-restore.md` in the GitHub repo.
+Muon on weight matrices, AdamW on embeddings and norms. WSD schedule with linear
+decay to zero over the final 45%. One RTX 5090, ~$46 of GPU time.
+
+Post-training: SFT on smol-smoltalk, then one DPO round on UltraFeedback.
+
+## Limitations
+
+- **English only**, 2048-token context, single seed.
+- **4-bit costs ~6% perplexity**, not the ~2.5% intended — quantisation-aware
+  training was built and validated, then crashed on activation and never ran.
+  The f16 files let you re-quantise without retraining.
+- **~48% of convolution channels are dead** (13.6M inert parameters). They
+  cannot be pruned at export: llama.cpp shape-checks those tensors against the
+  model width.
+- **Vocabulary is oversized** at 49,152 — inherited from a tokenizer chosen for
+  a distillation plan that was cancelled. Scaling laws suggest 24–32k here; it
+  costs 23% of parameters to a lookup table.
+- **Mixture skew 10.42** against a 10.0 pre-registered limit, from training
+  59.9B tokens on a 16.9B corpus.
+- It is a 150M model. It writes fluent, plausible text and gets many facts
+  wrong. The right reference class is GPT-2 124M.
+
+## Citation
+
+```bibtex
+@misc{koutsiaris2026daedalus,
+  title  = {Daedalus-150M: A Convolution--Attention Hybrid Designed for CPU Inference},
+  author = {Christos Koutsiaris},
+  year   = {2026},
+  url    = {https://github.com/unseen1980/daedalus}
+}
+```