morpheus-10M-v5

BabyLM 2026 Strict-Small (10M-word) submission: a ~135M-parameter Llama decoder trained from scratch with a curated morphological tokenizer.

Model

• Architecture: LlamaForCausalLM — 12 layers, hidden size 768, 12 attention heads, intermediate size 3072, rotary position embeddings, tied input/output embeddings
• Context length: 256 tokens
• Parameters: 135,050,496 (~135M)
• Vocabulary: 28,366 tokens (pruned morpheme inventory + character backoff)

Tokenizer

A curated, lemma-based morphological analyzer (built on MorphyNet + hand curation) that splits words into semantically-meaningful morphemes — e.g. unhappiness → ▁un happy ness, while keeping lexicalized wholes like business intact. The production vocabulary prunes morphemes attested fewer than three times and routes them through character backoff for out-of-vocabulary pieces. Self-contained: the analyzer, its resolved resource tables, and the vocabulary ship with the model, and it loads with trust_remote_code=True — no external dependencies at load time.

Training

• Objective: masked-next-token prediction (MNTP) at a 20% masking fraction (GPT-BERT-style 80/10/10 corruption; mask rate decays 0.30 → 0.15)
• Optimizer: Muon (lr 0.04) on 2D weight matrices + an AdamW auxiliary optimizer (lr 1e-3, weight decay 0.3) on embeddings and normalization parameters
• Schedule: cosine with 2% warmup; bf16 precision; 10 epochs over the 10M-word BabyLM corpus

Results (official babylm-eval pipeline)

• BLiMP: 72.52 on the full benchmark (72.92 on the fast_eval subset)
• BLiMP supplement: 58.3

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("juand-r/morpheus-10M-v5", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("juand-r/morpheus-10M-v5", trust_remote_code=True)

Data statement (datasheet)

Training corpus. We train on a cleaned version of the official BabyLM 2026 strict-small (10M) corpus — no other text corpus is used. Cleaning is deterministic and reproducible (scripts/clean_corpus.py --drop-childes-comments in the accompanying code release):

	Words (wc -w)
Official raw strict-small	10,000,019
Cleaned (what this model trained on)	9,732,593
Removed (≈2.7%)	267,426

What cleaning removes is transcription/annotation junk, not child-directed speech: 40,073 CHILDES transcriber-comment lines (the dominant removal), OpenSubtitles mojibake/OCR/ASS-tag/watermark artifacts (~2.5k lines), and Simple-Wikipedia HTML entities. Lines kept per source: bnc_spoken 65,220 · childes 491,232 · gutenberg 59,934 · open_subtitles 385,975 · simple_wiki 58,709 · switchboard 2,892.

Word-budget accounting (strict-small ≤ 10M). Training text = 9,732,593 words < 10M. Ten epochs = ~97.3M cumulative words seen < the 100M exposure cap. No synthetic data, no external model, and no distillation were used for this submission.

Tokenizer resources. The morphological analyzer is a lookup/rule system — nothing in it is trained on external text. It consults: (a) hand-curated morpheme tables written by the authors (28,544 entries); (b) the MorphyNet English inflectional/derivational lexicon, used in full (no frequency truncation in the submission pipeline): the shipped analyzer tables contain 187,132 form→segmentation entries plus a 617,042-item known-word list of single-word spellings (every lemma/form spelling in MorphyNet, consulted for is-this-a-word decisions during segmentation). These are dictionary-style resources consulted at tokenization time, fully documented in the accompanying paper; the only corpus statistics used to build the vocabulary come from the training corpus itself.

Effective lexicon usage. During training, the analyzer can only ever consult entries for words that occur in the training corpus (135,085 word types); all other entries are inert. The corpus-intersecting slice is: known-word list 59,285 (of 617,042), inflection entries 16,968 (of 187,132), curated entries 21,535 (of 28,544). Counted maximally strictly (every key and every segmentation-value word), this slice is 168,757 words, giving 9,732,593 + 168,757 = 9,901,350 < 10M even if lexicon annotations are charged against the word budget. (The intersecting key words are themselves corpus words, already counted once in the corpus total, so this is an upper bound.)

Intermediate checkpoints. Revisions chck_1M … chck_10M (every 1M words seen) and chck_20M … chck_100M (every 10M words seen) — the 19 checkpoints required for strict-small — are available as branches of this repository and load with revision="chck_<N>M".