SecludedCorner/bind1-babylm2026-strict-small

bind1 — Binding-Loop LM (24M), BabyLM 2026 Strict-Small entry

bind1 is the first generation of a planned architecture ladder (bind1 → bind2 → bind3) built around explicit role binding; the accompanying paper describes it as a microkernel loop LM. A 23.9M-parameter causal language model trained from scratch on the official BabyLM 2026 Strict-Small corpus (~10M words), within the 10-epoch exposure cap (150M tokens ≈ 9.2 epochs). The architecture is a weight-tied binding loop: an input block (3 layers), a core block (4 layers) unrolled T=3 times with an explicit role-binding write into a reserved 64-dim slice per iteration, and an output block (3 layers) — effective depth 18. A learned trust signal (v_gain) couples the model's self-consistency estimate to how much each iteration's re-binding is trusted during training.

• Entry id (internal): bind1_tt3_s0 (physical run name loop2_full), random seed 0 (BABYLM_SEED=0), frozen 2026-07-05 before any replication seeds were trained (no post-hoc seed selection).
• Track: Strict-Small. Backend: causal.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("REPO_ID", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("REPO_ID")

The model is fully self-contained (modeling_babylm.py ships in the repo); forward() runs the whole loop and returns standard CausalLMOutput(logits, loss). Empty-context, stateless across examples (competition-compliant: no external memory is used in training or evaluation).

Results (official babylm-eval, zero-shot, final checkpoint)

Task	Score
BLiMP	65.51
BLiMP supplement	58.35
COMPS	51.11
Entity tracking	39.55
ppl (held-out, internal)	9.4

GLUE fine-tuning (official protocol): BoolQ 67.9 · MNLI 45.1 · MRPC-F1 82.2 · QQP-F1 62.9 · RTE 54.7 · MultiRC collapsed to the majority class (F1 0.03 — shared with our size-matched monolith control; see paper for the tokenizer-separator analysis).

Growth checkpoints (competition requirement)

Branches chck_1M … chck_10M (every ~1M words through the first epoch) and chck_20M … chck_90M (every ~10M words thereafter; training ends at ≈92M words, so chck_100M does not exist). Checkpoints were saved on a fixed token cadence (1.63M tokens ≈ 1M words; 16M tokens ≈ 9.8M words), so nominal branch labels deviate from exact word counts by ≤1.7% — the mapping table:

branch	tokens	≈words		branch	tokens	≈words
chck_1M	1.6M	1.0M		chck_10M	16.3M	10.0M
chck_2M	3.3M	2.0M		chck_20M	32.3M	19.8M
chck_3M	4.9M	3.0M		chck_30M	48.3M	29.6M
chck_4M	6.5M	4.0M		chck_40M	64.3M	39.4M
chck_5M	8.2M	5.0M		chck_50M	80.3M	49.3M
chck_6M	9.8M	6.0M		chck_60M	96.3M	59.1M
chck_7M	11.4M	7.0M		chck_70M	112.3M	68.9M
chck_8M	13.0M	8.0M		chck_80M	128.3M	78.7M
chck_9M	14.7M	9.0M		chck_90M	144.3M	88.5M

Studying the learning curve (checkpoint branches)

The 18 growth checkpoints double as a learning-dynamics resource — each branch is a fully loadable model, so questions like "when does entity tracking emerge?" take one line per point:

model = AutoModelForCausalLM.from_pretrained(
    "SecludedCorner/bind1-babylm2026-strict-small",
    revision="chck_5M",   # any branch from the table above
    trust_remote_code=True)

The first-epoch cadence (every ~1M words, chck_1M…chck_10M) is deliberately dense for early-emergence studies; later checkpoints follow every ~10M words.

Training telemetry & audit tooling

• train_loop2_full.json — the complete training curve (1.8k records): loss/ppl plus the loop's full mechanism telemetry (verdict-to-trust gain v_gain, trust τ, verdict v, role entropy, per-iteration binding revision dBind, per-role priors). Every pathway of the loop is instrumented; this file is the primary evidence that the published mechanism was live during training (and it is how a silently-bypassed earlier variant was caught).
• audit_gate_bypass.py — a standalone audit script: point it at any exported model directory (python audit_gate_bypass.py <model_dir>) to check whether the loop actually contributes to the forward pass or has collapsed to a bypass. Reproduces the audit reported in the accompanying paper; runs on CPU in seconds.

Reproducibility

Trained on a single laptop GPU (RTX 4060, 8GB) in ~2h:

set BABYLM_SEED=0
python train_loop.py 384 3 4 3 3 256 16 150000000 loop2_full 16000000

(dim 384, in 3, core 4, out 3, T 3, seq 256, batch 16, 150M tokens, checkpoint every 16M tokens; first epoch additionally checkpointed every 1.63M tokens.) Tokenizer: 16k BPE trained on the Strict-Small corpus only. Full reproduction guide accompanies the paper.

Honest notes

• This is a single-seed entry. Replication seeds (trained after the freeze) show large seed variance on entity tracking — reported with full tables in the paper, not hidden.
• Inference-time iteration beyond the trained T does not help: the entity-tracking T-sweep degrades monotonically, and a per-item oracle analysis of adaptive-halting headroom is consistent with a 25% chance-level re-roll null. Both negatives are reported in the paper.
• The trust edge (v_gain) is causally inert at inference but training-critical: freezing it from scratch prevents the entity-tracking advantage from forming (training-scaffold evidence).

Citation

Yulin Yang (ORCID 0009-0007-4827-8449). A Microkernel Language Model: Reasoning as Mutually-Supporting Aggregates, and Why Its Parts Must Be Judged Together. BabyLM Challenge 2026 (Strict-Small track) submission.