miguelcsx/mosaic-memory-d384-bbpe16k

MOSAIC Memory — D384 BBPE16K

Top-three ablation from the controlled MOSAIC developmental-language-modeling study. The repository is a research release and has not yet been submitted to the BabyLM leaderboard.

Model summary

A bounded resource-memory objective under the common MOSAIC pretraining recipe, without lexical recombination.

Property	Value
Architecture	MOSAIC encoder, LTG-BERT-style relative attention
Parameters	33,320,644
Layers / hidden size	12 / 384
Attention heads	6
FFN size	1,280
Context length	512
Tokenizer	byte-level BPE, 16,384 tokens
Objective	whole-word masking + data2vec
Selected factors	uniform views + bounded memory
Unique training corpus	10M words
Total exposure	100M words, 10 passes

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "miguelcsx/mosaic-memory-d384-bbpe16k"
revision = "chck_100M"

tokenizer = AutoTokenizer.from_pretrained(repo_id, revision=revision)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    revision=revision,
    trust_remote_code=True,
)

Remote code is required because MOSAIC uses a custom Transformers architecture. Review tolm.py before loading it in a security-sensitive environment.

The canonical research code is maintained at github.com/miguelcsx/tolm. The bundled tolm.py is the exact, hash-pinned inference snapshot used by this release; the filename is retained solely for checkpoint compatibility.

Training recipe

• LAMB, peak learning rate 3.5e-3.
• Cosine schedule with 1.6% warmup and cooldown.
• BF16, 16,384 tokens per optimizer update.
• Whole-word masking with complementary coverage scheduling.
• Four-layer data2vec target, weight 0.5.
• EMA decay 0.9998.
• Dense document packing and a 512-token context.

The complete per-revision configuration is stored in training_manifest.json. File-level hashes for all revisions are recorded in release_manifest.json on main.

Checkpoint revisions

main and chck_100M contain identical final model weights.

Revision	Approximate exposure
chck_1M	1M words
chck_2M	2M words
chck_3M	3M words
chck_4M	4M words
chck_5M	5M words
chck_6M	6M words
chck_7M	7M words
chck_8M	8M words
chck_9M	9M words
chck_10M	10M words
chck_20M	20M words
chck_30M	30M words
chck_40M	40M words
chck_50M	50M words
chck_60M	60M words
chck_70M	70M words
chck_80M	80M words
chck_90M	90M words
chck_100M	100M words

Evaluation status

All 19 mandatory revisions have completed the internal fast screening suite. The final checkpoint has not yet completed the full fine-tuning evaluation, so this card deliberately makes no full-evaluation or leaderboard claim.

Reproducibility and provenance

Field	Value
Final model SHA-256	27796389f3045bcec7a4af49c1cecfac1484ef59883d18e58cb2c9205012384b
tolm.py SHA-256	5b87df8e506a408e9fd039e82b63ffcbdb844d3d7aa33a17c51fe8265cbadcff
Tokenizer SHA-256	9d7d221d3e8cdd0ac6f6385c91a943c1ab32ad1c8285332ea43a729cb42831b8
Evaluation backend	mntp
Evaluator source SHA-256	8a161c81c569cba37110f53b1a73d553cb1fbdedf60c062bec324f0cab9af32e
Evaluator commit at evaluation	3d57ddc8c40ee795c0b5e41b3a20251a9457a593
Equivalent clean XPU commit	e308acd4f462368fce723effa21b2e5c064e7734
Transformers / PyTorch	4.57.6 / 2.12.1+xpu
Maximum parity logit delta	0.0
Input/output embeddings tied	yes

The evaluator worktree was dirty when the original MOSAIC scores were generated because the XPU compatibility patch was not yet committed. The exact source tree is identified by its SHA-256 and was subsequently frozen in the clean equivalent commit listed above.

Limitations

• The model was trained on a small English developmental corpus and should not be treated as a general-purpose assistant.
• Fast screening scores are model-selection evidence, not held-out leaderboard results.
• mosaic_memory is one controlled study arm; comparisons should keep tokenizer, corpus, exposure budget, and evaluator provenance fixed.