README.md

---
license: apache-2.0
library_name: gguf
pipeline_tag: text-generation
language:
  - en
tags:
  - daedalus
  - cpu-inference
  - gguf
  - hybrid
  - conv-attention
base_model: Unseen1980/daedalus-checkpoints
---

# Daedalus-150M

A 150M-parameter language model built for **CPU inference**. Two thirds of its
layers are short convolutions with a fixed-size state instead of attention, so
decoding does not slow down as the context grows.

Trained from scratch on 59.9B tokens. Code and paper:
[unseen1980/daedalus](https://github.com/unseen1980/daedalus).

## Quick start

```bash
brew install llama.cpp   # or build from ggml-org/llama.cpp
hf download Unseen1980/daedalus-checkpoints instruct/model-q4_0.gguf --local-dir ./daedalus

llama-cli -m ./daedalus/instruct/model-q4_0.gguf -cnv \
  --temp 0.8 --top-p 0.9 --repeat-penalty 1.15
```

**Pass sampling flags.** llama.cpp defaults `--repeat-penalty` to 1.0, i.e. off,
and this model will loop on a repeated token without it.

## Files

| File | Size | What |
|---|---|---|
| `instruct/model-q4_0.gguf` | 102 MB | chat model, 4-bit — **start here** |
| `gguf/hero-base-q4_0.gguf` | 102 MB | base model, text completion |
| `gguf/instruct-f16.gguf` | 323 MB | instruct, f16 — for re-quantising |
| `gguf/hero-base-f16.gguf` | 323 MB | base, f16 |
| `hf/instruct/`, `hf/base/` | 321 MB | HF-format safetensors + tokenizer |
| `final/hero/checkpoint.pt` | 1.4 GB | base weights + optimizer state |
| `final/post-sft/final.pt` | 642 MB | instruct weights, full precision |

The **base** model deliberately carries no chat template. Giving one to a base
model makes llama.cpp wrap prompts in markup it never saw during training, which
produces fluent but unrelated output. Use plain prompts, or `llama-completion`.

## Results

Five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande, with
every peer re-scored on the same harness rather than quoted from its paper.

| Model | Training tokens | 5-task mean |
|---|---|---|
| **Daedalus-150M** | **59.9B** | **47.31** |
| MobileLLM-125M | 1T | 46.3 *(published)* |
| GPT-2 124M | — | 42.2 |
| OPT-125M | 180B | 42.1 |
| GPT-neo-125M | 300B | 41.9 |
| Pythia-160M | 300B | 41.0 |
| SmolLM2-135M | 2T | 51.2 |

Validation bits-per-byte **0.8685** over 645M held-out tokens.

SmolLM2-135M stays ahead on quality — conceded in advance. The trade this model
makes is speed.

## Speed

CPU decode, 4-bit, 8 threads, against a parameter-matched all-attention twin
trained on identical data:

| Context | Daedalus | Dense twin | Ratio |
|---|---|---|---|
| 0 | 1112 tok/s | 923 tok/s | 1.20× |
| 512 | 960 tok/s | 664 tok/s | 1.45× |
| **2048** | **739 tok/s** | **420 tok/s** | **1.76×** |

**The trend is the result.** At an empty context the hybrid has nothing to gain —
its advantage *is* the key–value cache it does not keep. Against an external
135M peer the same pattern reaches **2.08×** at 2048 tokens.

Per token of context this model reads 6,144 bytes of cache against a 24-layer
all-attention model's 12,288 — half. At 2048 tokens that is 12.6 MB re-read per
generated token instead of 25.2 MB.

## Architecture

```
18 blocks, d_model 768, vocab 49,152, context 2048

block:  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18
type:   C  C  C  C  A  C  C  A  C  A  C  A  C  A  C  C  A  C

A = full attention (6)      GQA, 12 query heads / 4 KV heads
C = short convolution (12)  depthwise, kernel 3, fixed 2-step state
```

Tied embeddings, 2048 FFN, RoPE θ=1e6. `Q4_0` chosen for ARM kernel speed rather
than its error curve.

## Training

59.9B tokens over a 16.9B-token corpus (~3.5 epochs, capped at 4 per source) of
public English data weighted toward educational text: FineWeb-Edu 37.5%,
DCLM-baseline 22.5%, Stack-Edu 9%, FinePDFs-Edu 8%, FinePhrase 7%,
Cosmopedia-v2 5%, FineMath + InfiWebMath 6%, FineWiki-en 3%, dialogue 2%.

Muon on weight matrices, AdamW on embeddings and norms. WSD schedule with linear
decay to zero over the final 45%. One RTX 5090, ~$46 of GPU time.

Post-training: SFT on smol-smoltalk, then one DPO round on UltraFeedback.

## Limitations

- **English only**, 2048-token context, single seed.
- **4-bit costs ~6% perplexity**, not the ~2.5% intended — quantisation-aware
  training was built and validated, then crashed on activation and never ran.
  The f16 files let you re-quantise without retraining.
- **~48% of convolution channels are dead** (13.6M inert parameters). They
  cannot be pruned at export: llama.cpp shape-checks those tensors against the
  model width.
- **Vocabulary is oversized** at 49,152 — inherited from a tokenizer chosen for
  a distillation plan that was cancelled. Scaling laws suggest 24–32k here; it
  costs 23% of parameters to a lookup table.
- **Mixture skew 10.42** against a 10.0 pre-registered limit, from training
  59.9B tokens on a 16.9B corpus.
- It is a 150M model. It writes fluent, plausible text and gets many facts
  wrong. The right reference class is GPT-2 124M.

## Citation

```bibtex
@misc{koutsiaris2026daedalus,
  title  = {Daedalus-150M: A Convolution--Attention Hybrid Designed for CPU Inference},
  author = {Christos Koutsiaris},
  year   = {2026},
  url    = {https://github.com/unseen1980/daedalus}
}
```