@GODELEV Hey, im not saying it definitly is, but it seems Benchmaxxed

I tested BananaMind and it gets 879, ThATS WORSE THAN My 3M parameter model.

And i run some other of my tools that look at the weights for patterns, they gave it a 85% benchmaxxing chance😔

Thanks a lot man, appreciate it!

Honestly, it’s mostly just a clean data mix and squeezing max efficiency per token. I blended ClimbMix with FineWiki, Cosmopedia, and a custom script I wrote to generate bug-free math word problems. Combined with a few architecture tweaks, it just learned super fast.

Thanks a lot man, appreciate it!

Honestly, it’s mostly just a clean data mix and squeezing max efficiency per token. I blended ClimbMix with FineWiki, Cosmopedia, and a custom script I wrote to generate bug-free math word problems. Combined with a few architecture tweaks, it just learned super fast.

Can you give me a link to ClimbMix? I want to reproduce the model

@kefir090 Im not saying your model is benchmaxxed, but it just seems odd here ill show you

Model Params Int Index HellaSwag ARC-Easy ARC-Chall PIQA ArithMark-3

1 SmolLM2-135M (HuggingFace) 135M 27.13 43.22 58.63 29.69 68.44 39.20
2 SmolLM-135M (HuggingFace) 135M 25.74 42.70 56.31 29.01 68.28 36.80
3 GPT-X2.5-135M (Axiomic Labs) 135M 25.17 40.57 51.81 29.18 69.42 38.40
4 MobileLLM-R1-140M-base (Meta) 140M 24.64 33.84 49.92 24.74 63.22 65.70
5 GPT-X2-125M (Axiomic Labs) 125M 23.36 40.41 51.47 27.82 67.30 37.20
6 BananaMind-2-Pro-Preview 138M 23.04 39.83 51.01 27.13 66.76 38.90
7 🔥 Lumen-118M-Base (ours) 117.5M 20.32 33.74 49.20 25.77 66.65 39.50
8 GPT-X-125M (Axiomic Labs) 125M 19.94 36.57 50.76 26.62 64.96 35.60
9 Supra2-100M-Base (SupraLabs) 101M 19.41 35.98 47.81 24.83 65.40 36.90
10 Supra2-100M-Instruct (SupraLabs) 101M 18.48 35.89 44.44 24.74 64.36 38.20
12 Museko-125M (TobiasLogic) 123M 16.88 33.26 48.06 25.68 63.22 34.60

But then one BananaMind Base Bench which you didnt include:

Lumne-118M-Base: Overall ELO: 879
BananaMind 2 Pro (releasing tomorrow):~1119
BananaMInd 2 Micro, which is only 3M Parameters: 890

@Banaxi-Tech ClimbMix is available on Hugging Face here: https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle

And no benchmaxxing or contamination here, the pretraining mixture uses standard public corpora (ClimbMix, FineWiki, Cosmopedia) plus a clean procedural math generator with random seeds, names, and numbers. All test splits were filtered against contamination.

Cosmopedia or cosmopedia v1

OK so i just checked, you didnt contaminate the model, but you kinda benchmaxxed it, the paper of ClimbMix litterally says THAT the Validation sets of PIQA, ARC, Hellaswag are in the data. This isnt contamination beause its the validation sets, But still benchmark maxxing or boosting the scores. @GODELEV And Base Bench shows that because it wasnt maxxed for that.

Still nice model! But i think you should say that in the README.

@Banaxi-Tech https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

Okay, I'll keep that in mind.

If you just add "ClimbMix includes the validation splits of ARC, PIQA and Hellaswag, some benchmark scores are affected by this" anywhere in the readme im fine!

@Banaxi-Tech Added the note about ClimbMix and validation splits to the README under both the Benchmark notes and Data Blend specs. Thanks for the heads up!

@kefir090 Thanks for adding that!

Has been removed from the leaderboard, ty for bringing it to my attention @Banaxi-Tech

Great work @Banaxi-Tech