instanton-baseline

instanton — a small model that trains almost instantaneously (~12 min on one A100). BabyLM Challenge 2026, Strict-Small track (<=10M words). Part of Reweighting Child-Directed and Conversational Data for Sample-Efficient BabyLM Pretraining.

• Architecture: GPT-2-style decoder-only transformer, from-scratch init (no pretrained weights): 6 layers, 512 hidden, 8 heads, 512-token context, 27.4M parameters (tied embeddings).
• Objective: causal language modeling.
• Tokenizer: byte-level BPE, 16k vocab, trained only on this run's <=10M-word training mixture.
• Training (v2 recipe): 10 epochs (~100M words exposure), packed 512-token sequences, batch 32 (no gradient accumulation, ~9.9k steps), lr 1e-3 cosine (5% warmup), weight decay 0.1, bf16, seed 42, ~12 minutes on one A100-80GB. The recipe was selected by a controlled sweep: 4x more optimizer steps at higher LR beat both a LLaMA-style architecture swap and a 97M-parameter capacity increase at this word budget.
• Training data: official BabyLM 2026 Strict-Small corpus (10,000,000 words), unmodified. Word counts use whitespace tokenization matching the official BabyLM 2026 dataset cards.

Intermediate checkpoints

As required by BabyLM 2026, checkpoints at every 1M words seen (to 10M) and every 10M (to 100M) are available as revisions chck_1M ... chck_100M:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("qyxu1994/instanton-baseline", revision="chck_5M")
tokenizer = AutoTokenizer.from_pretrained("qyxu1994/instanton-baseline")

Evaluation

Official babylm-eval pipeline, full zero-shot suite (entity tracking follows the leaderboard's 2026-07-07 "nothing"-answer-filtered protocol): BLiMP 65.93, BLiMP-supplement 54.33, EWoK 51.91, COMPS 52.00, (Super)GLUE 63.33.

Lineage

v2 of our submission family; v1 (same mixtures, shorter schedule) remains at qyxu1994/babylm-2026-strict-small-* for reproducibility.