miguelcsx/mosaic-views-d384-bbpe16k

Recombined curriculum checkpoint

This repository contains one trained member of a controlled five-run BabyLM Strict-Small family. The runs share source data, tokenizer, backbone, initialization seed, primary masked-language-model objective, main optimization schedule, and nominal exposure. There is one trained run per recipe, and added objectives require unequal training compute. Comparisons are descriptive.

Model and data

Property	Value
Architecture	custom LTG-BERT-style masked encoder with relative attention
Parameters	33,320,644
Layers / hidden size	12 / 384
Attention heads / FFN size	6 / 1,280
Context length	512
Tokenizer	byte-level BPE, 16,384 entries
Primary objective	complementary whole-word masking + data2vec
Source corpus	9,572,138 words
Deterministic replay	427,862 words
Corpus per pass	10,000,000 words
Nominal exposure	100,000,000 words

Training operation

Documents are ordered by subword tokens per word. The eligible fraction grows from 30% by 35 percentage points per epoch. On 40% of document visits, the first half is joined to the second half of a neighboring document that shares one of the source document's rare eligible tokens.

This is a corpus-presentation intervention. It is not an empirical variation-set manipulation and does not establish an effect on child language learning.

Shared training settings include LAMB with peak learning rate 0.0035, a cosine schedule with 1.6% warmup and cooldown, bfloat16, 16,384 tokens per update, data2vec weight 0.5, and EMA decay 0.9998.

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "miguelcsx/mosaic-views-d384-bbpe16k"

tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="main")
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    revision="main",
    trust_remote_code=True,
)

main contains the final 100M-word checkpoint. Earlier checkpoints are stored as chck_1M through chck_10M, then chck_20M through chck_90M. This repository has no chck_100M branch.

Remote code is required for the custom inference class. Review tolm.py before loading it in a security-sensitive environment. The canonical code is maintained at github.com/miguelcsx/tolm.

Evaluation and provenance

The required checkpoint revisions were evaluated with the same BabyLM fast suite used for the controlled family. These outputs are local development evidence, not a leaderboard rank or a superiority claim. Machine-readable configuration and file provenance are recorded in training_manifest.json and release_manifest.json.

Limitations

• This is one trained run, so it does not estimate training-run variance.
• The model is a small English masked LM, not a general-purpose assistant.
• The recipe label describes an implemented operation. It does not establish child-like input processing or a causal claim about children.