ecreeth/stream-mixer-33m-babylm

Stream Mixer 35.5M — BabyLM 2026 Strict

A ~35.5M-parameter Stream Mixer (linear-time, attention-free) trained on the BabyLM 2026 Strict 100M-word corpus for 10 epochs.

Model Details

• Architecture: Single-Scale Stream Mixer (20 layers; 8 streams × 32 dim; depthwise causal 1D conv; SwiGLU MLP; QK-Norm; no attention, no KV cache)
• Parameters: 35,481,760
• Layers: 20
• Hidden dim: 384
• Streams: 8 parallel memory streams (stream dim 32, 8 read heads)
• MLP hidden: 1024 (SwiGLU)
• Dropout: 0.02 - 0.08 (layer-wise split: half-rate mixer, 1.5x MLP)
• Vocab: 10,240 BPE (trained on BabyLM corpus)
• Context: 1024 tokens
• Training: 10 epochs, 1.9B tokens (53.5× params), Cosine decay schedule, MuonAdamW optimizer
• Chunked scan: C=512 (10% faster + 2× less memory than O(T²) at T=1024)

Comparison of Model Versions

Metric / Parameter	v8 (Multi-Scale)	v9 (Single-Scale)	v13	v14	v15	v16	v17	v18	v19	v20	v22	v23	v24	v25 (Current)
Architecture	Multi-Scale	Single-Scale	Single-Scale	Single-Scale	Single-Scale	Single-Scale	Single-Scale	Single-Scale	Alternating	Single-Scale	Single-Scale	Single-Scale	Single-Scale	Single-Scale
Parameters	27,544,336	28,662,144	28,363,896	27,602,040	27,602,040	36,802,680	33,500,280	32,532,640	32,025,720	35,481,760	35,512,576	43,407,680	41,791,680	35,481,760
Layers	8	16	20	20	20	20	20	20	20	20	16	20	24	20
Hidden dim	384	384	384	384	384	512	384	384	384	384	384	384	384	384
Streams	6	6	6	6	6	6	6	8	4/8	8	16	16	8	8
Vocab Size	16,384	16,384	10,240	10,240	10,240	10,240	10,240	10,240	10,240	10,240	10,240	10,240	10,240	10,240
Dropout	0.00	0.03	0.03	0.06	0.06	0.06	0.03	0.06	0.04 / 0.01	0.04	0.00	0.04	0.04	0.02-0.08
MLP Hidden	768	768	800	768	768	768	1024	896	768 / 1152	1024	1024	1024	1024	1024
Batch Size	128	256	512	512	576	576	576	512	576	512	512	512	512	512
Chunked Scan	No	No	No	No	C=512	C=512	C=512	C=512	C=512	C=512	C=512	C=512	C=512	C=512
Val Loss	3.4352	3.3719	3.2844	3.2711	3.2742	pending	3.2898	3.2773	3.3242	3.2891	3.3578	3.2789	3.2906	3.2703
Val Perplexity	31.04	29.13	26.69	26.33	26.42	pending	26.84	26.50	27.78	26.80	28.73	26.55	26.86	26.32

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ecreeth/stream-mixer-35m-babylm")
model = AutoModelForCausalLM.from_pretrained(
    "ecreeth/stream-mixer-35m-babylm",
    trust_remote_code=True,
)

inputs = tokenizer("The cat sat on the", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

BabyLM Challenge

• Track: Strict (100M words, 10 epochs max)
• Eval repo: babylm-eval
• Leaderboard: BabyLM-Leaderboard-2026

Results (zero-shot, causal, temperature 1.0)

Task	Baseline	v6	v9	v13	v14	v15	v16	v17	v18	v19	v20	v22	v23	v24	v25 (Current)
BLiMP	62.87	66.78	69.99	69.74	69.72	69.20	69.88	69.45	71.58	68.22	71.36	pending	68.94	69.29	68.76
EWOK (supplement)	49.64	54.33	51.54	54.46	54.46	56.00	53.55	55.38	53.29	54.86	53.94	pending	54.95	54.94	53.66
VQA (EWoK)	52.76	52.35	53.33	53.03	52.86	51.96	53.03	53.54	52.65	53.10	52.17	pending	53.01	52.32	51.33
Entity Tracking	17.90	17.41	17.45	39.67	39.75	22.60	33.07	41.96	26.72	41.39	40.74	pending	35.65	36.22	38.27
Comps	52.40	52.45	53.84	54.68	54.67	54.40	53.61	53.39	53.84	53.37	53.22	pending	54.18	53.54	53.61
Reading (eye tracking)	0.93	1.52	0.41	0.21	0.21	0.39	0.61	0.36	0.28	0.60	0.51	pending	0.34	0.38	0.71
Reading (self-paced)	0.14	0.02	0.53	0.00	0.00	0.08	0.67	0.27	0.01	0.29	0.00	pending	0.36	0.19	0.11

Results (fine-tuning, GLUE)

Task	Metric	Baseline	v6	v9	v13	v14	v15	v16	v17	v18	v19	v20	v22	v23	v24	v25 (Current)
BOOLQ	accuracy	63.8	64.59	64.46	64.22	64.22	pending	pending	63.91	65.14	pending	63.73	pending	64.16	pending	64.40
MULTIRC	accuracy	58.5	56.93	-	57.10	57.10	pending	pending	57.34	57.47	pending	57.05	pending	pending	pending	57.67
RTE	accuracy	61.2	59.71	-	51.08	51.08	pending	pending	67.31	53.96	pending	53.24	pending	pending	pending	58.99
WSC	accuracy	63.5	63.46	-	57.69	57.69	pending	pending	72.06	51.92	pending	57.69	pending	pending	pending	53.85
MRPC	f1	69.6	82.25	-	81.55	81.55	pending	pending	58.28	81.90	pending	81.79	pending	pending	pending	82.14
QQP	f1	69.6	54.98	-	55.71	55.71	pending	pending	45.99	59.07	pending	59.55	pending	pending	pending	57.33
MNLI	accuracy	43.6	44.68	-	45.29	45.29	pending	pending	pending	46.17	pending	45.46	pending	pending	pending	45.99

Architecture

The Stream Mixer replaces self-attention with a linear-time associative state recurrence:

1. Input tokens are processed by a depthwise causal 1D convolution (kernel_size=4) to inject local temporal context.
2. The model maintains M parallel streams, each of dimension D, updated by recurrence s[t] = α·s[t-1] + r·v with data-dependent decay α_t.
3. H multi-head sigmoid-gated reads with QK-Normalization (RMSNorm) query the associative memory.
4. A SwiGLU MLP feedforward layer processes the mixed representations.

This yields O(n) training complexity per token and constant-time O(1) decoding (with no KV cache) using model.step().

Training Details

Parameter	Value
Optimizer	MuonAdamW (Muon for 2D weights, AdamW for embeddings/biases)
LR schedule	Cosine Decay (warmup 500 steps, cosine decay to min_lr=5e-5)
Epochs	10 epochs
Peak LR	0.0080 (muon), 0.0005 (adamw)
Weight decay	0.12
Dropout	0.04
Batch size	64 × 8 accum = 512 effective × 1024 tokens (262k tokens/step)
Hidden dim	384
Streams	8 × 32 dim (8 read heads)
MLP hidden	1024
Total steps	3,623
Training time	10,005s (2h46m45s)
Throughput	~195k tok/s
GPU	NVIDIA A100 80GB
Best val loss	3.2891
Chunked scan	C=512 (10% faster + 2× less memory than O(T²))

Data cleaning

CHILDES speaker tags, bracket annotations, Wikipedia headers, subtitle formatting, HTML tags filtered

Links

• BabyLM Challenge 2026
• BabyLM Eval