remg1997/modernbert-small-phase6-composition-babylm2026

ModernBERT-Small (Compositional Byte-N-Gram Embeddings) — BabyLM 2026 Strict-Small

This model is a ModernBERT-Small masked language model (RoPE, GeGLU, alternating local/global attention, 384 hidden size, 16 layers, 6 attention heads) trained from scratch on the BabyLM 2026 Strict-Small 10M-word corpus, as part of an ablation study on parameter-efficient token embedding layers for developmentally-plausible pretraining under the BabyLM Challenge's strict-small compute and data budget.

Embedding design

Standard transformer token embedding tables scale as vocab_size x hidden_size, which for this model's 30,522-token vocabulary and 384 hidden size would be an 11.7M-parameter dense lookup table. This checkpoint replaces that table entirely with a compositional byte-n-gram embedding, inspired by fastText's subword hashing trick: each vocabulary token is decomposed into its raw UTF-8 byte sequence, every byte n-gram (n = 1 to 4) is extracted from it and hashed into one of 8,192 shared buckets, and the token's representation is the mean of its buckets' embeddings (a 128-dimensional EmbeddingBag table, ~1.0M parameters total) rather than a dedicated per-token row. This composed vector is then projected up to the model's 384-dimensional hidden size. The same shared n-gram bucket table is used to produce the MLM output head's logits (tie_word_embeddings=true is required for this embedding type), so the entire vocabulary's token representations are generated algorithmically from ~1M shared subword-hash parameters instead of ~11.7M independent per-token ones. This particular checkpoint uses no token-specific residual on top of the composed representation (compositional_residual_mode=none) -- it is the "composition-only" control in a broader sweep that also tests frequency-gated residual variants.

This is one variant in a broader comparison of embedding-layer parameterizations (dense, linear and MLP-style factorization, tensor-train decomposition, deterministic Fourier expansion, and compositional/frequency-adaptive variants) evaluated under identical data, tokenizer, optimizer, and training budget, to isolate the effect of the embedding layer's parameterization on downstream BabyLM evaluation performance.

Training data

BabyLM 2026 Strict-Small corpus (~10M words), tokenized with a byte-level BPE tokenizer trained on the same corpus (vocab size 30,522). No external data, synthetic augmentation, or human annotation beyond the corpus as officially released.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained(
    "remg1997/modernbert-small-phase6-composition-babylm2026",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "remg1997/modernbert-small-phase6-composition-babylm2026"
)

Each chck_{N}M branch of this repository corresponds to a BabyLM-Challenge compliance checkpoint (one per N million words of training data seen); main points at the final, fully-trained checkpoint.

Evaluation

Evaluated with the official babylm-eval harness (BLiMP, EWoK, entity tracking, COMPS, Global PIQA, reading-time correlation, GLUE/SuperGLUE fine-tuning, and Age-of-Acquisition word-surprisal correlation) under the strict-small track.