TinyBrainBot-3M-Loop

A 2.96M-parameter language model that reuses its layers. Four transformer blocks are run three times in a loop, so every token passes through 12 layers of computation while only 4 layers of weights are stored. Trained from scratch on 5.5B tokens.

Benchmark (0-shot, full set)	acc_norm
Open SLM Leaderboard Intelligence Index	20.12
(raw acc)	20.58
HellaSwag	27.36
(raw acc)	26.94
ARC-Easy	30.81
(raw acc)	33.67
ARC-Challenge	22.95
(raw acc)	18.00
PIQA	56.04
(raw acc)	57.67
ArithMark-3.0	89.3
(raw acc)	89.7

lm-eval 0.4.13 in float32 for HellaSwag, ARC and PIQA; the official bencharithmark-3.py (float32) for ArithMark-3.0.

Arithmetic on wording it has never seen

ArithMark-3.0 is easy to overfit: a tiny model can learn one benchmark's sentence patterns instead of arithmetic. So we tested on held-out sets in the same continuation format (about 1,600 items over 29 grade-school skills). Their stories were written by models that wrote none of our training data (Gemma-3-27B and Llama-3.3-70B), Python set every number and answer, and a separate model re-solved every item. Scoring is the same as ArithMark-3.0 (4 endings, length-normalized).

The traps versions use the same stories, but the wrong endings are typical mistakes instead of nearby numbers: a number copied from the story, the result of the wrong operation, and the answer plus or minus 1 or 10. A model that echoes or pattern-matches numbers falls below chance (25%) there.

Model	ArithMark-3.0 (our run)	Llama-written	Llama, traps	Gemma-written	Gemma, traps
TinyBrainBot-3M-Loop (2.96M)	89.3	92.8	90.9	69.3	58.1
Qwen2.5-0.5B	60.9	68.3	64.5	63.8	57.9
SmolLM2-360M	58.9	59.6	52.4	57.4	43.8
Pascaline-3M	69.2	47.1	22.6	39.0	11.2
SmolLM2-135M	38.8	33.2		36.2
GPT-S3-3M	38.9	33.0	9.1	36.2	9.9

AM3 numbers in this table are all our own runs of the official bencharithmark-3.py (float32) on the current ArithMark-3.0 files, so every model is scored the same way. They can differ slightly from the values listed on the leaderboard (e.g. SmolLM2-360M: 58.9 here, 59.6 listed).

It keeps its accuracy when the wording changes and when the wrong answers are plausible mistakes. Its weak spot is length: the Gemma-written stories are longer and chattier (about 200 characters against 120), and it handles plain, compact word problems much better than long ones with extra detail.

How it works

• Recurrent depth. 4 shared blocks (width 256, 4 attention heads of 64 with 2 KV heads, SwiGLU 560, RoPE, RMSNorm) are applied 3 times. Each pass adds a learned loop embedding and a gated copy of the input embedding. Training sampled 2 to 4 loops per step, and you can change the count at inference (model.model.eval_loops = 4); 3 is best.
• Factorized embedding. The 4,104-token vocabulary uses 96-dimensional embeddings projected up to 256 (and down again before the tied output layer), which frees about 630k parameters for the 4th block.
• Numbers. A units-first digit tokenizer writes every number as a length token plus its digits, smallest place first, and a digit-place embedding tells the model which place each digit is in.

Training

5.5B tokens, constant learning rate then a decay over the last 15%.

Part	Share	What
Q&A and facts	25%	our distilled Q&A, short facts, explanations
Arithmetic	20%	word problems: about 150k different story wordings written by ling-3.0-flash and Mistral-Small-3.2, each checked by solving it, with every number and answer filled in by Python; plus our own relation (more/fewer than) and story generators
Synthetic textbooks	15%	Cosmopedia (Stanford, OpenStax)
Stories	15%	TinyStories
Commonsense	10%	Cosmopedia WikiHow, our everyday-commonsense prose
Web	10%	FineWeb-Edu
Reference	5%	Wikipedia

The decay used more arithmetic (45%) and WikiHow (20%). No benchmark training or test data was used, and none of the arithmetic generators copy ArithMark's sentences.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-3m-loop"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
ids = tok("Mia had 46 stickers. She gave 18 to her brother, so now Mia has", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=8, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:]))

trust_remote_code=True is required (custom architecture and tokenizer). There is no KV cache: generation recomputes the prefix, which is fast at this size. There is no GGUF, because llama.cpp does not support the looped architecture.

To reproduce the scores:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-loop,dtype=float32,trust_remote_code=True --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-loop --dtype float32

Limitations

• A 3M base model: it writes simple English and makes up facts. It is not a chat model.
• Arithmetic is strongest in short word problems; long, chatty setups lower its accuracy.
• 1,024-token context.