MinimaLabs/KLM-1.5

KLM 1.5

KLM 1.5 is a 99M-parameter base language model. It is a new pretrain compared to KLM 1.1 which continued training KeyLM-75M whereas KLM 1.5 has a new architecture and was trained from scratch on 100B tokens.

What's new

• Hybrid layers. KLM 1.5 mixes Gated DeltaNet-2 and full attention layers at a 3:1 ratio. The Gated DeltaNet-2 layers carry a fixed-size recurrent memory through the sequence instead of attending to every earlier token.
• Block Attention Residuals (Kimi Team, 2026). Each layer reads a learned mix of the embeddings and the outputs of every earlier layer, instead of adding everything into one running residual sum.
• Exact n-gram memory. The 125,000 most frequent bigrams and 38,000 most frequent trigrams in the training data each have their own learned vector, which is looked up exactly and fed into the model next to the token embeddings. The memory holds 5.4M parameters.
• A small tokenizer. The vocabulary has 4,100 tokens and splits every number into single digits. Since the embeddings are tied, they only cost 2.1M parameters, which leaves almost all of the model for the layers.

Benchmarks

All scores are zero-shot with length-normalized accuracy.

• ARC, HellaSwag and PIQA were scored using lm-eval 0.4.12.
• ArithMark-3 was scored using the official ArithMark-3.0 script.
• Peer scores are taken from the Open SLM leaderboard as of 2026-10-22.

On that leaderboard, KLM 1.5's Intelligence Index is 30.26 which puts it first among models under 100M parameters and second under 150M.

Each chart's axis starts at random-guess accuracy, which is 25% for most benchmarks and 50% for PIQA, so the height of a bar shows how far a model scores above chance.

Model	Params	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	ArithMark-3
KLM 1.5	99.0M	59.68%	32.68%	43.74%	68.55%	48.20%
KLM 1.1	88.1M	44.36%	25.17%	33.65%	64.80%	35.80%
GPT-X3-150M	149M	59.64%	31.31%	46.16%	71.33%	50.90%
SmolLM2-135M	135M	58.63%	29.69%	43.22%	68.44%	39.20%
LFM-2.5-230M-Base	230M	59.72%	34.98%	39.29%	63.38%	36.60%
void.1	90.1M	47.31%	28.16%	38.68%	67.46%	44.80%

Training

KLM 1.5 was trained on 2,048-token sequences with 128 sequences per step, in three stages.

1. Superposition warm-up for 70,000 steps, using Token-Superposition Training. Each input position is the average of 7 consecutive tokens, and the model learns to predict the next 7 tokens as an unordered group. This lets the warm-up cover 128B tokens of ClimbMix for the compute of about 18B.
2. Stable training on 80B tokens of next-token prediction at a constant learning rate.
3. Decay on 20B tokens of a mix with more math and Q&A, while the learning rate falls to zero along a 1−√t curve.

The weight matrices were trained with MuonEq-R at a learning rate of 0.01, and the embeddings, norms and gates with AdamW at 3e-3, in bf16. Each 2,048-token sequence packs several documents, and the model never looks across a document boundary since attention is masked there, and the recurrent state and n-gram history are reset.

The data mix shifted between the stable and decay stages as follows.

Data	Sources	Stable	Decay
Web text	ClimbMix, TxT360-v2, UltraX	56%	26%
Q&A	TxT360-v2 Q&A, Dolmino Q&A and flashcards	13%	23%
Textbooks	Cosmopedia v2, Cosmopedia WikiHow	12%	14%
Math	OpenMathInstruct-2, UltraData-Math, Math-Reasoning, Dolmino TinyMATH	11%	20%
Science web	Dolmino	3%	8%
Commonsense	DCLM-baseline filtered for physical commonsense, ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ, QASC	3%	8%
Reasoning rewrites	Pretrain-Behaviors	2%	1%

Architecture

Field	Value
Parameters	98,975,308, of which 5,411,832 are the n-gram memory
Layers	24, Gated DeltaNet-2 and attention at 3:1
Hidden size	512
Gated DeltaNet-2	4 heads × 128
Attention	8 query heads, 4 KV heads, head size 64, RoPE, QK-norm, output gate
Feed-forward	SwiGLU, hidden size 1,408
Context length	2,048 tokens
Vocabulary	4,100 tokens, tied embeddings

Intended use

KLM 1.5 is a base model. It continues text but has not been trained to follow instructions or chat, so it is meant to be fine-tuned for your own use case. The tokenizer already has <|im_start|>, <|im_end|>, <think> and </think> as single tokens, so they are ready to use in chat and reasoning fine-tunes.

Usage

The model code is plain PyTorch, so KLM 1.5 runs on a CPU as well as on any GPU without extra packages beyond Transformers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MinimaLabs/KLM-1.5"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).to(device)

inputs = tokenizer("The three primary colors are", return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• The model's knowledge is thin, and the facts it states are often wrong.
• It only works in English and has a 2,048-token context window.
• It has no safety alignment, so add your own filtering before putting it in front of users.

License

KLM 1.5 is released under the Apache-2.0 license, which you can find in LICENSE.