ecreeth/BabyLM-2026-Baseline-Stream-Mixer-27M

Stream Mixer 27M — BabyLM 2026 Strict

A ~27M-parameter Stream Mixer (linear-time, attention-free) trained on the BabyLM 2026 Strict 100M-word corpus for 5 epochs.

Model Details

• Architecture: Multi-Scale Stream Mixer (fine-scale streams + stride-4 coarse-scale streams with read-gate fusion; no attention, no KV cache)
• Parameters: 27,532,032
• Layers: 8
• Hidden dim: 384
• Streams: 48, stream dim 96, 6 read heads (state capacity: 4,608 floats/layer)
• Vocab: 16,384 BPE (trained on BabyLM corpus)
• Context: 1,024 tokens
• Training: 5 epochs, ~950M tokens (34.5× params), WSD schedule, MuonAdamW optimizer

Comparison vs Previous Version (streammixer-26m-babylm)

The Multi-Scale version implements a hierarchical sequence mixer rebalancing parameters from the feedforward layers (MLP expansion reduced to 2.0×) into the mixer state capacity, adding a coarse-scale stride-4 hierarchy.

Metric / Parameter	Previous Version (streammixer-26m-babylm)	New Version (Multi-Scale)	Difference / Improvement
Architecture	Vanilla Single-Scale StreamMixer	Multi-Scale StreamMixer	Hierarchical Coarse+Fine Stream Fusion
Parameters	26,247,616	27,532,032	+1.28M params (+4.9%)
Validation Loss	2.97	2.8805	−0.0895 nats (~8.6% Perplexity drop)
Parallel Streams	32	48	+50% Stream count
Stream Dim	64	96	+50% Stream width
State Capacity ($M \times D$)	2,048 floats / layer	4,608 floats / layer	2.25× more working memory capacity
Read Heads	4	6	+50% Read expressive heads
MLP Hidden Dim	1024 (2.66× expansion)	768 (2.0× expansion)	Params reallocated from MLP to Streams

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ecreeth/BabyLM-2026-Baseline-Stream-Mixer-27M")
model = AutoModelForCausalLM.from_pretrained(
    "ecreeth/BabyLM-2026-Baseline-Stream-Mixer-27M",
    trust_remote_code=True,
)

inputs = tokenizer("The cat sat on the", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

BabyLM Challenge

• Track: Strict (100M words, 10 epochs max)
• Eval repo: babylm-eval
• Leaderboard: BabyLM-Leaderboard-2026

Results (zero-shot, causal, temperature 1.0)

Task	Score (Vanilla baseline)	Score (Multi-Scale v6)
BLiMP	62.87	66.78
EWOK (supplement)	49.64	54.33
VQA (EWoK)	52.76	52.35
Entity Tracking	17.90	17.41
Comps	52.40	52.45
Reading (eye tracking)	0.93	1.52
Reading (self-paced)	0.14	0.02

Results (fine-tuning, GLUE)

Task	Metric	Score (Vanilla baseline)	Score (Multi-Scale v6)
BOOLQ	accuracy	63.8	64.59
MULTIRC	accuracy	58.5	56.93
RTE	accuracy	61.2	59.71
WSC	accuracy	63.5	63.46
MRPC	f1	69.6	82.25
QQP	f1	69.6	54.98
MNLI	accuracy	43.6	44.68

Zero-shot tasks measure linguistic knowledge; fine-tuning tasks measure transfer learning to downstream classification. Evaluations are performed using the external babylm-eval pipeline on HF-converted checkpoints.

Architecture

The Stream Mixer replaces self-attention with linear-time stream mixing:

1. Input tokens are embedded and fed into n_streams parallel streams
2. Streams are mixed via learned query/read-head projections (no attention matrix)
3. A lightweight feedforward layer processes the mixed streams
4. Output is projected back to vocabulary logits

This gives O(n) complexity per token (vs O(n²) for Transformers) and no KV cache — generation uses model.step() for token-by-token decoding.

Training Details

Parameter	Value
Optimizer	MuonAdamW (Muon for 2D weights, AdamW for embeddings/biases)
LR schedule	Warmup-Stable-Decay (85% stable)
Epochs	5 epochs (max allowed is 10)
Peak LR	5e-3 (muon), 5e-4 (adamw)
Weight decay	0.1
Batch size	128 × 1,024 tokens (auto-scaled to GPU)
Total steps	7,247
GPU	NVIDIA A100 / H100 GPU (~60 min on Colab)
Val loss	2.8805
Data cleaning	CHILDES speaker tags, bracket annotations, Wikipedia headers, subtitle formatting, HTML tags filtered

Links

• BabyLM Challenge 2026
• BabyLM Eval