instanton model card

README.md

@@ -0,0 +1,53 @@
+---
+language: en
+license: mit
+tags:
+- babylm
+- babylm-2026
+- strict-small
+- gpt2
+---
+
+# instanton-baseline
+
+**instanton** — a small model that trains almost instantaneously (~12 min on one
+A100). BabyLM Challenge 2026, **Strict-Small track** (<=10M words). Part of
+*Reweighting Child-Directed and Conversational Data for Sample-Efficient
+BabyLM Pretraining*.
+
+- **Architecture:** GPT-2-style decoder-only transformer, from-scratch init
+  (no pretrained weights): 6 layers, 512 hidden, 8 heads, 512-token context,
+  27.4M parameters (tied embeddings).
+- **Objective:** causal language modeling.
+- **Tokenizer:** byte-level BPE, 16k vocab, trained **only** on this run's
+  <=10M-word training mixture.
+- **Training (v2 recipe):** 10 epochs (~100M words exposure), packed 512-token
+  sequences, batch 32 (no gradient accumulation, ~9.9k steps), lr 1e-3 cosine
+  (5% warmup), weight decay 0.1, bf16, seed 42, ~12 minutes on one A100-80GB.
+  The recipe was selected by a controlled sweep: 4x more optimizer steps at
+  higher LR beat both a LLaMA-style architecture swap and a 97M-parameter
+  capacity increase at this word budget.
+- **Training data:** official BabyLM 2026 Strict-Small corpus (10,000,000 words), unmodified. Word counts use whitespace tokenization
+  matching the official BabyLM 2026 dataset cards.
+
+## Intermediate checkpoints
+
+As required by BabyLM 2026, checkpoints at every 1M words seen (to 10M) and
+every 10M (to 100M) are available as revisions `chck_1M` ... `chck_100M`:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained("qyxu1994/instanton-baseline", revision="chck_5M")
+tokenizer = AutoTokenizer.from_pretrained("qyxu1994/instanton-baseline")
+```
+
+## Evaluation
+
+Official [babylm-eval](https://github.com/babylm-org/babylm-eval) pipeline,
+full zero-shot suite (entity tracking follows the leaderboard's 2026-07-07
+"nothing"-answer-filtered protocol): BLiMP 65.93, BLiMP-supplement 54.33, EWoK 51.91, COMPS 52.00, (Super)GLUE 63.33.
+
+## Lineage
+
+v2 of our submission family; v1 (same mixtures, shorter schedule) remains at
+`qyxu1994/babylm-2026-strict-small-*` for reproducibility.