yashmahe2018/rwkv7-halfclm-strict-small

RWKV-7 Half-CLM — BabyLM 2026 Strict-Small

A ~28M-parameter RWKV-7 recurrent (subquadratic) language model trained from scratch on the BabyLM 2026 Strict-Small corpus (14.87M tokens, 32K BPE). Causal CLM, zero auxiliary objectives. Inference is a single forward (CLM) pass.

This is the primary competitive submission for the BabyLM 2026 Strict-Small track and the headline model from Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models.

Method

Identical RWKV-7 architecture trained with halved CLM exposure: odd optimizer steps advance the learning-rate schedule without applying a gradient update ("step-skipping"). This reduced update pressure prevents the late-training collapse that the full-exposure baseline suffers, and does so without any auxiliary objectives (no MLM, MTP, or n-gram heads).

Performance (official BabyLM 2026 pipeline, strict-small)

Scores below are from a clean, end-to-end re-run of the official evaluation pipeline (evaluation_pipeline.sentence_zero_shot / reading / finetune), zero manual editing of predictions.

All scores come from a single, self-consistent re-run (one set of predictions, no cherry-picking across runs):

Metric	Score
BLiMP (filtered)	68.70
BLiMP Supplement	57.24
EWoK	1.53 (form-degenerate; shared by all non-baseline teams)
Entity Tracking	19.98 (near-chance; see note)
COMPS	52.55
Reading	0.86
GLUE	58.72
AoA	~0
Text Average (8-col)	32.45

Note on Entity Tracking. Run through the official pipeline, these weights score 19.98 on the full ET set, and the number is deterministic: three independent re-runs give 19.98, and applying first-index versus random tiebreaks to the same captured log-probs gives 19.98 either way (100 random draws, std 0.00). Internal evaluation corroborates it (~18.6 at step 18k). An earlier standing submission recorded ET ~42, but that figure is not reproducible from these released weights and we do not carry it forward. We report 19.98, the model's actual output.

Architecture

• RWKV-7 (RWKV7ForCausalLM), 8 layers, n_embd 384, vocab 32,000
• Chunked WKV-7 kernel (diagonal-only within-chunk decay, chunk_size 64) — disclosed approximation
• Pure-PyTorch modeling for HF eval compatibility (ported from the MLX training implementation)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "yashmahe2018/rwkv7-halfclm-strict-small", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "yashmahe2018/rwkv7-halfclm-strict-small", trust_remote_code=True
)

Citation

From the BabyLM 2026 submission Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models.