Add littlerock-1M base checkpoint (OpenCerebral)
Adds opencerebral/littlerock-1M, a public 1,002,720-parameter model pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning. Loads with plain transformers (stock LlamaForCausalLM, no trust_remote_code).
This supersedes #72, which submitted a finetuned checkpoint. That one is withdrawn: it scored 35.5% on your private held-out set but 21.5% on the variety set, so its arithmetic does not survive rephrasing. Thanks to @danaxiomic and the team for catching it and for explaining the three-set methodology.
Evaluation: lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via bencharithmark-3.py (float32, default acc_norm).
| Model | ARC-Easy | ARC-Challenge | HellaSwag | PIQA | ArithMark-3 | Int Index |
|---|---|---|---|---|---|---|
| littlerock-1M | 30.98 | 21.76 | 26.43 | 52.23 | 25.20 | 2.29 |
ArithMark-3 is at chance, which is the honest number for a 1M-parameter base model with no arithmetic training.
Architecture: 6 layers, d=96, 6 heads, SwiGLU 224, RoPE, RMSNorm, tied embeddings, 512 context, 4,096-token custom BPE (a stock 50k vocab would cost several times the whole parameter budget at this width). Trained in 1h49m on one RTX 3060.