TinyBrainBot-3M-Base

Superseded by TinyBrainBot-3M-v2-Base: a balanced general 3M (Int 10.38) whose arithmetic holds up on held-out wordings.

A 2.94M-parameter language model trained from scratch on 5.55B tokens, as a plain Llama (LlamaForCausalLM): no custom code.

Scores vs other models of 3M parameters or fewer

Zero-shot, full test sets, acc_norm. The (raw acc) row underneath shows our plain accuracy. The Intelligence Index is the Open SLM Leaderboard's, and the other models' numbers come from that leaderboard.

Model	Params	Int Index	HellaSwag	ARC-E	ARC-C	PIQA	ArithMark-3
TinyBrainBot-3M-Base (this model)	2.94M	11.33	26.94	33.12	23.04	55.98	51.20
(raw acc)		11.55	26.63	34.72	18.94	57.73	50.50
GPT-S3-3M	2.97M	9.61	27.68	34.76	23.12	57.02	39.10
Tokle-3M	2.91M	8.91	27.20	34.85	23.98	55.01	40.80
pulvis-v2	2.96M	8.62	27.93	31.86	22.78	56.86	37.40
Purrence-3M	2.99M	8.49	27.81	34.55	23.63	56.64	34.80

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nkthebass/tinybrainbot-3m-base")
model = AutoModelForCausalLM.from_pretrained("nkthebass/tinybrainbot-3m-base")
ids = tok("Ava had 23 apples. Ava got 19 more. Now Ava has", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=6, do_sample=False)[0]))
# -> ... 42 apples

There is also an F16 GGUF (tinybrainbot-3m-base-F16.gguf, 6 MB) for llama.cpp, LM Studio and Jan. A Q8_0 file isn't possible because the hidden size (144) is not a multiple of 32.

This is a base model: it continues text and does not chat. At this size, expect short, simple text and grade-school arithmetic in familiar phrasings.

Model

Parameters	2,942,928 total (2.35M non-embedding + 0.59M tied embedding)
Shape	hidden 144, 10 layers, 3 query heads of 48 with 1 KV head (GQA), SwiGLU 416, RMSNorm, RoPE
Vocabulary	4,096-token byte-level BPE trained for this model: digits split one per token, whitespace kept exactly, no unknown tokens
Context	1,024 tokens

Training

5.55B tokens: 4.72B of pretraining, then a 0.83B-token decay.

Pretraining	Share
Q&A and facts (our distilled plain-text sets)	25%
Grade-school story arithmetic (generated by our own programs)	20%
Cosmopedia Stanford + OpenStax	15%
TinyStories	15%
Everyday / how-to (Cosmopedia WikiHow + distilled)	10%
FineWeb-Edu	10%
Wikipedia leads	5%

Decay (0.83B tokens)	Share
The pretraining sets, re-weighted toward Q&A, facts and commonsense	50%
Generated grade-school arithmetic in many phrasings	35%
Cosmopedia WikiHow	15%

The arithmetic is written by our own programs: every number is computed in Python and nothing is written by an LLM. It covers adding, taking away, multi-step problems, groups and sharing, money, time, measurement, area and perimeter, fractions, patterns and tallies, each in many different sentence shapes. No benchmark training or test data was used.

Recipe: AdamW (betas 0.9/0.95, weight decay 0.1), peak LR 4e-3 with a warmup-stable-decay schedule, 131k tokens per step, fp16 mixed precision, trained on 2× V100.

How this checkpoint was chosen. We ran four decay branches with different arithmetic data from the checkpoint at step 36,000. They scored between 8.80 and 11.33 on the Index. This is the best one, and the published weights are that branch's final checkpoint, not a merge.

One thing we learned. Our first version learned its arithmetic from a single sentence pattern. In that pattern it solved 2-digit sums with carrying 99% of the time, but in other wordings it was near chance, and it scored 33.6 on ArithMark-3. At this size, the skill gets tied to the wording, and spreading the same arithmetic over many phrasings is what fixed it.

Evaluation

Run in float32 with lm-eval 0.4.13 (full sets, 0-shot) and the official ArithMark-3 script:

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-3m-base,dtype=float32 --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-3m-base --dtype float32

Limitations

At 3M parameters this is a toy-scale model. It knows few facts, loses coherence after a few sentences, and its arithmetic only works reliably in common phrasings. HellaSwag and ARC are close to chance for every model this size, so the gaps between models there are small.