MinimaLabs/min-spark-1.6

min-spark 1.6

min-spark 1.6 is a 9.96M-parameter language model with native effort levels. It scores 11.08 on the Open SLM leaderboard's Intelligence Index, the highest of any model under 10M parameters, ahead of ForgePlex-M2.1-10M (10.70) and min-spark 1.5 (9.79).

What's new in 1.6

min-spark 1.6 uses the min-spark 1.5 architecture: a prelude block, five shared transformer blocks run two to four times in a row, and a coda, with a separate LoRA adapter for each pass on every projection of the shared blocks. The one architectural change is the adapter rank, raised from 24 to 33. That adds 0.6M parameters, and rank 33 is the largest that keeps the model under 10M.

The rest of the change is in training. 1.6 is trained from scratch for 20B tokens, against 1.5's 2.0B, on a new data mix. Before that, a superposition warm-up reads text in groups of eight tokens for 13,500 steps, covering 28B tokens of text for the compute of 3.5B.

Evaluation

All scores are zero-shot and length-normalized. ARC, HellaSwag and PIQA use lm-eval 0.4.12; ArithMark-3 uses the official ArithMark-3.0 script. min-spark 1.6 is scored at each benchmark's best effort level.

Model	Params	Training tokens	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	ArithMark-3
min-spark 1.6	9.96M	20B + 28B warm-up	38.30%	22.44%	28.52%	59.14%	36.90%
ForgePlex-M2.1-10M	9.99M		36.45%	22.87%	29.09%	59.14%	35.50%
heron1i-flash-lite	9.92M	24B	36.20%	24.15%	28.36%	58.54%	35.60%
min-spark 1.5	9.36M	2.0B	38.64%	23.55%	28.27%	57.34%	34.90%
GPT-S3-3M	2.97M	26.5B	34.76%	23.12%	27.68%	57.02%	39.10%

Effort levels

Effort levels set how many passes the model makes through its shared core. Large reasoning models usually do this by changing a thinking-token budget. min-spark changes the computation inside the network instead.

Effort	Passes	Use
low	2	Fastest completion
medium	3 (default)	General use; the best single setting
high	4	Experimental: one pass more than the model was trained with. It still gives the best ArithMark-3 score

Effort	ARC-Easy	ARC-Challenge	HellaSwag	PIQA	ArithMark-3
min-spark 1.6 low	37.50%	22.44%	28.52%	58.60%	35.90%
min-spark 1.6 medium	38.30%	21.67%	28.24%	59.14%	36.00%
min-spark 1.6 high	37.63%	22.01%	28.48%	58.65%	36.90%
best per task	38.30%	22.44%	28.52%	59.14%	36.90%

Usage

min-spark 1.6 loads through Transformers with remote code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "MinimaLabs/min-spark-1.6",
    trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(
    "MinimaLabs/min-spark-1.6",
    trust_remote_code=True,
)

inputs = tokenizer("The meaning of life is", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, effort="medium", max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

effort works the same way with the pipeline API. To skip Transformers, use the bundled script:

python generate.py -p "The meaning of life is" -e medium

Training

min-spark 1.6 was trained from scratch in four phases:

Phase	Steps	Text read	What the model learns
Bracket matching	96	25M tokens	Next-token prediction on synthetic nested brackets
Superposition warm-up	13,500	28.3B tokens of ClimbMix	Predicting the next group of 8 tokens
Stable	61,152	16.0B tokens	Next-token prediction at full learning rate
Decay	15,052	3.9B tokens	Next-token prediction while the learning rate falls to 10%

Superposition warm-up

The warm-up follows Token-Superposition Training (Peng, Gigant & Quesnelle, 2026). Each input position is the average of 8 consecutive token embeddings, and the model predicts which 8 tokens make up the next group, ignoring their order. A step costs the same as a normal step but covers 8 times as much text. After 13,500 steps the model switches to normal next-token training, with no change to the architecture, the tokenizer or inference. The paper's smallest model has 270M parameters, 27 times larger than this one. We did not run a controlled comparison, so we cannot say how much of 1.6's gain comes from the warm-up.

Held-out loss fell from 3.07 after the first 1B tokens to 2.87 at the end, and most of the final drop came in the decay.

Field	Value
Steps	89,800 at 512 sequences × 512 tokens
Precision	bf16 autocast
Optimizer	Muon (matrices, learning rate 0.01) + NAdamW (other parameters), weight decay 0.01
Learning-rate schedule	2,000-step warmup, constant, then linear decay to 10% over the final 15,052 steps
Objective	Next-token loss at 3 passes, plus 0.1 × the loss at 2 passes and 0.1 × a term that keeps the 2-pass state close to the 3-pass one
Attention masking	Intra-document

Data

Source	Stable	Decay	License
ClimbMix web text, read from the OptimalScale copy (also the whole warm-up)	40%	20%	CC BY-NC 4.0
TxT360-v2 web pages with reading-comprehension Q&A	12%	8%	CC BY 4.0
TxT360-v2 web text	8%		CC BY 4.0
Cosmopedia v2 synthetic textbooks	15%	15%	ODC-BY
Commonsense: DCLM-baseline filtered for physical commonsense, plus wikiHow, ATOMIC 2020, ANLI, GLUCOSE, CREAK, SciQ and QASC	10%	20%	CC BY 4.0, CC BY-NC 4.0 (ANLI), CC BY-NC 3.0 (SciQ), CC BY-NC-SA 3.0 (wikiHow), MIT (CREAK)
OpenMathInstruct-2 problems with solutions	6%	12%	CC BY 4.0
Dolmino science web crawl	4%	8%	ODC-BY
Dolmino passages with questions (wiki_to_rcqa)	1%	4%	ODC-BY
Dolmino synthetic Q&A (nemotron-synth-qa)	2%	5%	ODC-BY
Dolmino multiple-choice flashcards (reddit_to_flashcards)	1%	4%	ODC-BY
Dolmino grade-school math dialogues (tinymath-mind)	1%	4%	ODC-BY

Every source was checked against the five evaluation sets: any document sharing a 13-word sequence with an evaluation item was dropped.

Architecture

Field	Value
Parameters	9,956,756
Architecture	Meiosis (tied-embedding looped decoder)
Vocabulary	4,096-token byte-level BPE
Embedding width	288
Heads	6 query · 2 KV (GQA)
FFN hidden size	832
Blocks	1 prelude · 5 shared body blocks · 1 coda · final RMSNorm
Per-pass adapters	LoRA rank 33 on attention QKV, attention output, FFN gate/up and FFN down
Context window	512 tokens
Effort (loop count)	low = 2 · medium = 3 · high = 4 (trained up to 3)

Reproducing the evaluation

python run_lmeval.py --effort low,medium,high
python bencharithmark-3.py --model MinimaLabs/min-spark-1.6 --dtype float32

The first command scores ARC, HellaSwag and PIQA at each effort and prints each task's best (add --limit N for a quick run). The second is the official ArithMark-3 script, which runs at the default medium effort. ArithMark-3's best score comes at high: to reproduce it, download the repo, set "effort": "high" in its config.json, and pass the local path to --model.

Limitations

• Base model only. min-spark 1.6 completes text. It is not instruction-tuned and does not hold a conversation.
• Small model. Its world knowledge is thin and its facts are often wrong. Do not rely on it for factual answers or math.
• English only, with a 512-token context window. All results were measured within it.
• No safety alignment. Add your own filtering before any user-facing use.

License

Apache-2.0. See LICENSE. Training-data licenses are listed in the Data table.