license: apache-2.0
library_name: gguf
pipeline_tag: text-generation
language:
- en
tags:
- daedalus
- cpu-inference
- gguf
- hybrid
- conv-attention
base_model: Unseen1980/daedalus-checkpoints
Daedalus-150M
A 150M-parameter language model built for CPU inference. Two thirds of its layers are short convolutions with a fixed-size state instead of attention, so decoding does not slow down as the context grows.
Trained from scratch on 59.9B tokens. Code and paper: unseen1980/daedalus.
Quick start
brew install llama.cpp # or build from ggml-org/llama.cpp
hf download Unseen1980/daedalus-checkpoints instruct/model-q4_0.gguf --local-dir ./daedalus
llama-cli -m ./daedalus/instruct/model-q4_0.gguf -cnv \
--temp 0.8 --top-p 0.9 --repeat-penalty 1.15
Pass sampling flags. llama.cpp defaults --repeat-penalty to 1.0, i.e. off,
and this model will loop on a repeated token without it.
Files
| File | Size | What |
|---|---|---|
instruct/model-q4_0.gguf |
102 MB | chat model, 4-bit — start here |
gguf/hero-base-q4_0.gguf |
102 MB | base model, text completion |
gguf/instruct-f16.gguf |
323 MB | instruct, f16 — for re-quantising |
gguf/hero-base-f16.gguf |
323 MB | base, f16 |
hf/instruct/, hf/base/ |
321 MB | HF-format safetensors + tokenizer |
final/hero/checkpoint.pt |
1.4 GB | base weights + optimizer state |
final/post-sft/final.pt |
642 MB | instruct weights, full precision |
The base model deliberately carries no chat template. Giving one to a base
model makes llama.cpp wrap prompts in markup it never saw during training, which
produces fluent but unrelated output. Use plain prompts, or llama-completion.
Results
Five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande, with every peer re-scored on the same harness rather than quoted from its paper.
| Model | Training tokens | 5-task mean |
|---|---|---|
| Daedalus-150M | 59.9B | 47.31 |
| MobileLLM-125M | 1T | 46.3 (published) |
| GPT-2 124M | — | 42.2 |
| OPT-125M | 180B | 42.1 |
| GPT-neo-125M | 300B | 41.9 |
| Pythia-160M | 300B | 41.0 |
| SmolLM2-135M | 2T | 51.2 |
Validation bits-per-byte 0.8685 over 645M held-out tokens.
SmolLM2-135M stays ahead on quality — conceded in advance. The trade this model makes is speed.
Speed
CPU decode, 4-bit, 8 threads, against a parameter-matched all-attention twin trained on identical data:
| Context | Daedalus | Dense twin | Ratio |
|---|---|---|---|
| 0 | 1112 tok/s | 923 tok/s | 1.20× |
| 512 | 960 tok/s | 664 tok/s | 1.45× |
| 2048 | 739 tok/s | 420 tok/s | 1.76× |
The trend is the result. At an empty context the hybrid has nothing to gain — its advantage is the key–value cache it does not keep. Against an external 135M peer the same pattern reaches 2.08× at 2048 tokens.
Per token of context this model reads 6,144 bytes of cache against a 24-layer all-attention model's 12,288 — half. At 2048 tokens that is 12.6 MB re-read per generated token instead of 25.2 MB.
Architecture
18 blocks, d_model 768, vocab 49,152, context 2048
block: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
type: C C C C A C C A C A C A C A C C A C
A = full attention (6) GQA, 12 query heads / 4 KV heads
C = short convolution (12) depthwise, kernel 3, fixed 2-step state
Tied embeddings, 2048 FFN, RoPE θ=1e6. Q4_0 chosen for ARM kernel speed rather
than its error curve.
Training
59.9B tokens over a 16.9B-token corpus (~3.5 epochs, capped at 4 per source) of public English data weighted toward educational text: FineWeb-Edu 37.5%, DCLM-baseline 22.5%, Stack-Edu 9%, FinePDFs-Edu 8%, FinePhrase 7%, Cosmopedia-v2 5%, FineMath + InfiWebMath 6%, FineWiki-en 3%, dialogue 2%.
Muon on weight matrices, AdamW on embeddings and norms. WSD schedule with linear decay to zero over the final 45%. One RTX 5090, ~$46 of GPU time.
Post-training: SFT on smol-smoltalk, then one DPO round on UltraFeedback.
Limitations
- English only, 2048-token context, single seed.
- 4-bit costs ~6% perplexity, not the ~2.5% intended — quantisation-aware training was built and validated, then crashed on activation and never ran. The f16 files let you re-quantise without retraining.
- ~48% of convolution channels are dead (13.6M inert parameters). They cannot be pruned at export: llama.cpp shape-checks those tensors against the model width.
- Vocabulary is oversized at 49,152 — inherited from a tokenizer chosen for a distillation plan that was cancelled. Scaling laws suggest 24–32k here; it costs 23% of parameters to a lookup table.
- Mixture skew 10.42 against a 10.0 pre-registered limit, from training 59.9B tokens on a 16.9B corpus.
- It is a 150M model. It writes fluent, plausible text and gets many facts wrong. The right reference class is GPT-2 124M.
Citation
@misc{koutsiaris2026daedalus,
title = {Daedalus-150M: A Convolution--Attention Hybrid Designed for CPU Inference},
author = {Christos Koutsiaris},
year = {2026},
url = {https://github.com/unseen1980/daedalus}
}