Spaces:

---

AxiomicLabs

/

Open_SLM_Leaderboard

like 78

Running

Adds opencerebral/littlerock-1M, a public 1,002,720-parameter Llama-architecture model pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline) and then finetuned on public train splits. It loads with plain transformers (no trust_remote_code).

Evaluation: lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via your own bencharithmark-3.py (float32, default acc_norm).

The ArithMark-3 number is high for this size, so the provenance is up front rather than buried in the card.

• ArithMark-3 contributed zero training data. It has no train split -- its 1,000 items are the benchmark. Numeracy comes from elementary word problems generated from templates written independently for this project; ArithMark's own templates were not consulted.
• No scoring split was trained on. Finetuning used the train splits of PIQA, ARC-Easy and ARC-Challenge (train+validation), HellaSwag, OpenBookQA and SciQ. Scoring is on test/validation, which those do not overlap.
• Contamination guard. Every training string was checked against an index of all 16,428 items across all five scoring splits -- exact context, exact context+ending for every choice, and 13-gram overlap. It is validated with controls: it catches 200/200 real ArithMark items, 200/200 HellaSwag, 200/200 PIQA and 100/100 case/punctuation-edited variants, while flagging 0/3000 of the synthetic arithmetic. On the real corpus it found and dropped 221 genuine collisions -- HellaSwag's train split overlaps its own validation split via shared wikiHow sources, and OpenBookQA overlaps ARC.
• No checkpoint was selected on a benchmark. Selection used a held-out dev set: slices of the train splits never touched by training, plus arithmetic drawn from template families withheld from training, so the dev signal measures cross-phrasing transfer rather than template recall. The real suite was run only after selection was finished.

The per-topic breakdown from your script is the check on all of that:

Performance is high on single-operation topics the generator covers and at or below chance (25.0) on topics it does not. Contamination or memorisation would raise every topic uniformly. Two-step chaining fails, which is a real limit: this model does one arithmetic operation, not two.

For reference, the same model before the finetune scores HellaSwag 26.37 / ARC-E 30.64 / ARC-C 21.50 / PIQA 52.29 / ArithMark-3 25.90 (Int Index 2.36). Essentially all of the gain is arithmetic; the language benchmarks are near a capacity wall at 1M parameters. Happy to submit the base checkpoint instead, or in addition, if you would prefer that.

Full methodology is on the model card. We have tried to follow the leaderboard's rules as we understand them, and everything above is disclosed so you can judge for yourselves -- if you determine that any part of this falls outside what you intend for the leaderboard, that is your call to make and we will accept it; we are happy to withdraw the submission or replace it with the base model.