op-meco-scratch-c0 โ€” from-scratch metadata-conditioned pretrain (scratch-c0)

OFFICIAL-PRETRAIN from-scratch testbed (user ask 2026-08-16): conditioning from TOKEN ZERO, the intervention point the literature says matters. OLMo-2 architecture d512/8L/8H, tied embeddings, stock OLMo tokenizer (~77M params, ~25M non-embedding); 90,478 labeled dolma docs @ dataset revision 48f4f8aa, ~4 epochs, lr 4e-4 cosine, wd 0.1, block 1024, bs 64, seed 17.

Arm: bare documents throughout (control)

Siblings: op-meco-scratch-c0 / -plain / -tagged. pretrain_stats.json has the full accounting. Part of the meco/ route (branch meco).

Downloads last month
68
Safetensors
Model size
77M params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support