Liodon AI - SLM-10M: How a 10M Model Topped the SLM sub-10M Leaderboard

We just released SLM-10M, a 9.97M-parameter causal language model that leads the sub-10M tier on the Open SLM Leaderboard with an average score of 32.38%.

At 10M parameters, it's 12× smaller than GPT-2 (124M) and 13× smaller than SmolLM2-135M — yet it holds its own against models 5-10× its size on several benchmarks.

This post covers the architecture, training recipe, and design decisions that made it possible.

---

The Results

Zero-shot evaluation on five benchmarks:

The key result: SLM-10M beats Pythia-31M despite having less than a third of the parameters. It also comes within striking distance of GPT-2 on HellaSwag and ArithMark-2.0.

---

Architecture: Standing on the Shoulders of Giants

SLM-10M follows the modern SLM recipe established by GPT-X2, Qwen3, and Gemma2:

Why These Choices?

GQA (Grouped Query Attention) — Using 8 query heads with only 2 KV heads cuts KV cache memory by 4× compared to MHA, which matters enormously at the 10M scale where every byte of activation counts.

QK-Norm — Normalizing queries and keys before attention prevents logit explosion during training. At 10M parameters, training instability can destroy weeks of compute. QK-Norm is cheap insurance.

RoPE with θ=100,000 — A high RoPE base frequency gives the model better extrapolation beyond its 1,024-token training context, which helps on benchmarks with longer inputs.

SwiGLU — The gated FFN variant consistently outperforms standard GeGLU and MLP at small scales. The 640-dim intermediate (2.5× hidden) follows the LLaMA/Qwen ratio.

RMSNorm with fp32 upcast — Normalization in fp32 prevents precision loss in bfloat16 training, which is critical when your model has only 256 hidden dimensions.

Scaled residual init — Residual projections (o_proj, FFN down) are initialized with 0.02 / √(2 × 12) to keep the residual stream variance bounded across 12 layers. Without this, the model either collapses or diverges in the first few thousand steps.

---

Training: 25B Tokens, One GPU

Data Mix

The data mix prioritizes quality over quantity. FineWeb-Edu provides broad knowledge, Cosmopedia-v2 adds structured reasoning content, FineWeb-HQ contributes clean prose, and FineMath boosts arithmetic capability.

Optimizer and Schedule

The warmup-stable-decay schedule is critical: the model needs a long stable phase to consolidate knowledge before the cosine tail fine-tunes the representations. Cutting the stable phase short causes the model to forget what it learned.

Z-Loss

We use logit z-loss (weight=1e-4) during the first 31B tokens to prevent attention logit explosion, then disable it. The model is stable enough after 31B tokens that z-loss only adds noise.

Hardware

Trained on a single NVIDIA GB10 in bfloat16 with torch.compile. Throughput: ~2-3M tokens/second. Total training time: ~8-12 hours for 25B tokens.

---

What Works at 10M (and What Doesn't)

What Works

• Log-likelihood ranking — SLM-10M excels at multiple-choice QA where it scores candidate answers and picks the highest. This is its primary use case.
• Pattern recognition — The model learns arithmetic progressions, grammatical patterns, and basic logical structure.
• Perplexity evaluation — Useful for ranking document relevance or candidate continuations.

What Doesn't

• Open-ended generation — At 10M parameters with an 8,192-token vocabulary, free-form text generation is not fluent. The model is a research artifact for benchmarking, not a chat model.
• Instruction following — No instruction tuning was applied. The model responds to prompts in the style of its pretraining data.
• Complex reasoning — Multi-step reasoning, code generation, and advanced math are beyond its capacity.

---

The Bigger Picture

SLM-10M is part of a growing ecosystem of tiny models competing on the Open SLM Leaderboard. The leaderboard currently tracks 50+ models from 20+ organizations, ranging from 1.3M to 150M parameters.

What we're seeing is that architecture matters more than size at the sub-100M scale. A well-designed 10M model with GQA, QK-Norm, SwiGLU, and RoPE can outperform a poorly-designed 30M model with a vanilla Transformer architecture.

The key insight: every component at the 10M scale must earn its place. There's no room for architectural bloat. Each layer, each head, each normalization must contribute meaningfully to the model's capability.

---

Try It

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import torch.nn.functional as F

model = AutoModelForCausalLM.from_pretrained(
    "liodon-ai/slm-10m",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("liodon-ai/slm-10m", trust_remote_code=True)

def score(context, completion):
    full = tokenizer.encode(context + completion, return_tensors="pt").to("cuda")
    ctx_len = len(tokenizer.encode(context, add_special_tokens=False))
    with torch.no_grad():
        logits = model(full).logits[0]
    return -F.cross_entropy(logits[ctx_len - 1:-1], full[0, ctx_len:]).item()

context = "Which is an example of a renewable energy resource? Answer:"
choices = [" biomass", " coal", " gas", " oil"]
scores = [score(context, c) for c in choices]
best = choices[scores.index(max(scores))]
print(f"Best answer: {best.strip()}")
# → Best answer: biomass

---

What's Next

• SLM-50M — Scaling up to 50M parameters with the same architecture
• Instruction tuning — Adding chat and instruction-following capability
• SLM-Bench — Our new benchmark specifically designed for sub-10M models (dataset, lm-eval PR)
• Architecture ablations — Systematic study of which components matter most at each scale

---

Links

• Model · Leaderboard · SLM-Bench · GitHub

SLM-10M is released under Apache-2.0 by Liodon AI — an independent U.S.-based research lab focused on advancing small language models and efficient deep learning infrastructure.