TinyBrainBot-3M-Loop-v3

A 2.96M-parameter math-first language model that reuses its layers. Four transformer blocks are run three times in a loop, so every token passes through 12 layers of computation while only 4 layers of weights are stored. Trained from scratch on 6.5B tokens. Its arithmetic carries over to bare expressions, order of operations, numbers written in words, large numbers, thousands separators and money, not only small numbers in word-problem stories.

Benchmark (0-shot, full set)	acc_norm
Open SLM Leaderboard Intelligence Index	19.44
(raw acc)	19.92
HellaSwag	27.04
(raw acc)	26.88
ARC-Easy	31.86
(raw acc)	32.91
ARC-Challenge	22.01
(raw acc)	18.17
PIQA	53.97
(raw acc)	55.88
ArithMark-3.0	91.6
(raw acc)	91.6

lm-eval 0.4.13 in float32 for HellaSwag, ARC and PIQA; the official bencharithmark-3.py (float32) for ArithMark-3.0.

Arithmetic beyond one benchmark

Every column below is a zero-shot, log-likelihood multiple-choice math test, all scored by us with the official bencharithmark-3.py scoring in float32, so every model is measured the same way (numbers can differ slightly from leaderboard listings).

Model	Params	ArithMark-3	AM3, words	AM3, bare equations	ArithMark-2	ArithMark-1	Held-out stories	Held-out, large numbers	Held-out, commas	Held-out, money
TinyBrainBot-3M-Loop-v3	2.96M	91.6	53.9	97.3	83.9	35.8	88.4	93.1	91.4	93.7
TinyBrainBot-3M-Loop-v2	2.96M	90.7	55.0	96.8	83.8	36.1	80.1	45.3	39.0	40.0
TinyBrainBot-3M-Loop	2.96M	89.3	27.3	88.2	58.1	29.4	81.0	39.2	37.7	42.4
Inferno-Exp-1.7	394M	85.9	59.1	86.9	59.9	47.9	74.0	62.7	74.4	70.8
Pascaline-3M	3M	69.2	27.6	99.7	66.5	36.2	43.0	39.9	37.8	42.3
SmolLM2-135M	135M	38.8	34.2	38.6	33.2	33.1	34.7	37.4	39.5	38.6
GPT-S3-3M	2.97M	38.9	25.8	36.2	31.0	31.1	34.6	34.3	34.4	35.9

• Numbers in words and bare equations are ArithMark-3's own problems with only the surface changed.
• ArithMark-2 uses its official metric (raw log-likelihood); its official benchmark_arithmark-2.0.py gives the same results.
• Held-out stories: about 1,600 ArithMark-style problems written by Gemma-3-27B and Llama-3.3-70B, models that wrote none of our training data, with every number and answer set by Python.
• Held-out, large numbers / commas / money: the 958 held-out sum and difference stories with every amount scaled into 4-6 digit numbers, thousands separators ("74,986") or dollars and cents ("$70.67"). The previous versions were near chance (25%) on these.
• ArithMark-1: its answer key mostly evaluates expressions strictly left to right (e.g. 14 * -17 + -1 * -18 = 4302), while ArithMark-2 and ordinary arithmetic use operator precedence (that expression is -220). This model follows precedence, so it scores lower on ArithMark-1's longer expressions (61% on its two-operand stage, falling with each extra operator).

How it works

• Recurrent depth. 4 shared blocks (width 256, 4 attention heads of 64 with 2 KV heads, SwiGLU 560, RoPE, RMSNorm) are applied 3 times. Each pass adds a learned loop embedding and a gated copy of the input embedding. Training sampled 2 to 4 loops per step, and you can change the count at inference (model.model.eval_loops = 4); 3 is best.
• Factorized embedding. The 4,104-token vocabulary uses 96-dimensional embeddings projected up to 256 (and down again before the tied output layer), which frees about 630k parameters for the 4th block.
• Numbers. A units-first digit tokenizer writes every number as a length token plus its digits, smallest place first, and a digit-place embedding tells the model which place each digit is in.

Training

Stage 1 (4.46B tokens): the TinyBrainBot-3M-Loop run up to the start of its learning-rate decay.

Part	Share	What
Q&A and facts	25%	our distilled Q&A, short facts, explanations
Arithmetic	20%	word problems: about 150k different story wordings written by ling-3.0-flash and Mistral-Small-3.2, each checked by solving it, with every number and answer filled in by Python; plus our own relation (more/fewer than) and story generators
Synthetic textbooks	15%	Cosmopedia (Stanford, OpenStax)
Stories	15%	TinyStories
Commonsense	10%	Cosmopedia WikiHow, our everyday-commonsense prose
Web	10%	FineWeb-Edu
Reference	5%	Wikipedia

Stage 2 (2.0B tokens, math-first): constant learning rate for 1.33B tokens, then a decay to zero over the last 0.67B.

Part	Share	What
Number formats	22%	the verified story wordings with amounts scaled to large numbers, thousands separators, money with cents and decimals, about 30% with unrelated sentences around the problem; plus bare expressions in the same formats
Story arithmetic	16%	the same verified wordings, with a quarter of the problems' numbers written in words
Bare arithmetic	16%	our generator: expressions with order of operations, parentheses, negative numbers and 2-5 operands, in about 19 forms ("43 + 19 = 62", "What is ...?", "Compute ...:")
Number words	7%	words to digits and back, arithmetic in words, counting, comparisons
Relations and stories	9%	our relation and story generators
Q&A and facts	20%	as in stage 1
Cosmopedia, FineWeb-Edu	10%	as in stage 1

No benchmark training or test data was used, and none of our generators copy any benchmark's problems, code or sentences.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "nkthebass/tinybrainbot-3m-loop-v3"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
ids = tok("Mia had 46 stickers. She gave 18 to her brother, so now Mia has", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=8, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:]))

trust_remote_code=True is required (custom architecture and tokenizer). There is no KV cache: generation recomputes the prefix, which is fast at this size. Padded batches work on either side (the model honors attention_mask). There is no GGUF, because llama.cpp does not support the looped architecture.

To reproduce the scores:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-loop-v3,dtype=float32,trust_remote_code=True --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-loop-v3 --dtype float32

Limitations

• A 3M base model: it writes simple English and makes up facts. It is not a chat model.
• Arithmetic is strongest on short problems; long, chatty setups and long left-to-right chains lower its accuracy.
• 1,024-token context.