littlerock-1M

A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning.

Architecturally a stock Llama, so it loads with plain transformers — no trust_remote_code, no custom modelling file:

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/littlerock-1M")
model = AutoModelForCausalLM.from_pretrained("opencerebral/littlerock-1M")

Results

lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via AxiomicLabs' bencharithmark-3.py (float32, default acc_norm).

Benchmark	Score
HellaSwag	26.43
ARC-Easy	30.98
ARC-Challenge	21.76
PIQA	52.23
ArithMark-3	25.20
Avg	32.56
Intelligence Index	2.29

ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic.

A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all.

Architecture

Parameters	1,002,720
Non-embedding	609,504
Layers	6
Hidden size	96
Heads	6 (head dim 16)
FFN hidden (SwiGLU)	224
Context	512
Vocab	4,096
Position encoding	RoPE
Norm	RMSNorm
Embeddings	tied

The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer's embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level BPE with byte fallback (no UNK is possible), fitted on 400k documents sampled from this model's own pretraining mixture.

Training

Tokens	5,000,000,000
Mixture	40% FineWeb-Edu, 60% DCLM-baseline
Steps	19,073
Batch	262,144 tokens/step
Optimizer	AdamW (0.9, 0.95), wd 0.1
LR	3e-3 cosine → 3e-4, 50M-token warmup
Precision	bf16 autocast, fp32 master weights
Final val loss	3.6251 (ppl 37.6)
Hardware	1x RTX 3060, 1h 49m

Data was materialised to a fixed local binary before training (no streaming), with documents EOS-separated, and the token budget enforced exactly at 5,000,000,000.

Limitations

This is a ~1M-parameter base model. It is a next-token predictor with no instruction tuning and no alignment, it has very limited world knowledge, and it will produce fluent but frequently false text. It exists to measure what a 1M-parameter budget can do.

Sample (prompt in italics): The capital of France is a national world of fashion, a country and many other nationwide nationality. In this case, the public has been banned in the early 17th century…