Add cagliostro-v1 (157.4M, Index 19.32)
Adds cagliostro-v1, a 157.4M base model trained from scratch on 36B tokens.
Datdanboi25 invited this submission on Discord and confirmed the size is fine.
Two lines changed in index.html: a benchlabs entry in the org registry and the
model appended to the bottom of MODELS, as the comment there asks. No existing
entry is touched. The chart colour #c026d3 is unused by any other org, easy to
change if you would rather it were something else.
Numbers are lm-evaluation-harness 0.4.12, zero-shot, acc_norm, full test sets,
bfloat16, batch 32, on a single RTX 4060 Ti.
hellaswag 36.90
arc_easy 48.91
arc_challenge 26.11
piqa 65.61
arithmark3 32.80
arithmark2 26.72
Intelligence Index works out to 19.32.
Because ArithMark is not an lm-eval task, I wrote a task config for it and
calibrated the whole setup against your published GPT-2 row before trusting any
of it. Running GPT-2 through the identical pipeline gives:
task mine yours diff
hellaswag 31.23 31.26 -0.03
arc_easy 39.73 39.35 +0.38
arc_challenge 23.04 22.35 +0.69
piqa 62.51 62.08 +0.43
arithmark3 35.80 35.70 +0.10
arithmark2 25.56 26.48 -0.92
The ArithMark config that reproduces your number prepends activity_label to
ctx, HellaSwag style. Scoring ctx alone gives GPT-2 28.30 instead of 35.70, so
if you ever see someone report a low ArithMark that is probably why.
GPT-2 Index comes out 0.44 high through my pipeline versus your published row,
so a re-run on your side would likely land cagliostro nearer 18.9 than 19.32.
Either way it slots in at rank 10. Happy to adjust the row to whatever your own
run produces.
Training data was decontaminated against HellaSwag, ARC, PIQA and ArithMark-3
before pretraining with a 13-gram overlap filter built from 1,407,822 n-grams
over 58,032 eval fields.