Unseen1980/daedalus-150m-instruct

Daedalus-150M — Instruct

A 150M-parameter language model built for CPU inference. Full attention is kept in only 6 of its 18 layers; the other 12 use short convolutions whose memory is two timesteps wide however long the conversation gets. Decoding therefore does not slow down as context grows.

Trained from scratch on 59.9B tokens, then instruction-tuned (SFT on smol-smoltalk + one DPO round on UltraFeedback).

• GGUF builds and full checkpoints: Unseen1980/daedalus-checkpoints
• Code and paper: unseen1980/daedalus

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Unseen1980/daedalus-150m-instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

out = model.generate(inputs, max_new_tokens=96, temperature=0.8,
                     top_p=0.9, repetition_penalty=1.15, do_sample=True)
print(tok.decode(out[0], skip_special_tokens=True))

Set repetition_penalty. Without it this model can lock onto a word and repeat it until it runs out of tokens.

For CPU deployment, use the 4-bit GGUF (102 MB) from the checkpoints repository rather than these weights.

Results

Five-task mean over HellaSwag, ARC-Easy, PIQA, OpenBookQA and WinoGrande, with every peer re-scored on the same harness rather than quoted from its paper:

Model	Training tokens	5-task
Daedalus-150M	59.9B	47.31
MobileLLM-125M	1T	46.3 (published)
GPT-2 124M	—	42.2
OPT-125M	180B	42.1
GPT-neo-125M	300B	41.9
Pythia-160M	300B	41.0

Validation bits-per-byte 0.8685. CPU decode ~440 tokens/second, and 1.76× faster than a same-size all-attention model at 2048 tokens of context — an advantage that grows with context rather than staying constant.

Limitations

It is a 150M model. It writes fluent, plausible text and gets plenty of facts wrong; the fair comparison is GPT-2 124M, which it beats. Short factual answers and explanations work best. Open-ended creative writing drifts after a few lines. English only, 2048-token context, single seed.

The 4-bit build costs about 6% perplexity — quantisation-aware training was built but did not run. Roughly 48% of the convolution channels are inert and cannot be pruned, and the 49,152-entry vocabulary is larger than this model size warrants. All three are documented in the paper.