Spaces:

---

AxiomicLabs

/

Open_SLM_Leaderboard

like 77

Running

Merged, Ty for your submission! Great model, will chuck you a follow to keep track of your stuff (plus first model smaller then gpt-x to beat it)

@Datdanboi25 I need to very importantly inform you, Lumen trained on the Validation sets of PIQA, Hellaswag and ARC bringing up its score. We tested it on BananaMind Base bench where it didnt train and the score was below my 3M parameter model.
Important:
It only trained on the Validation sets, Not the benchmarks i think but lm_eval may also evaluate the validation sets.

I dont know if this is allowed on your leaderboard because its only validation.

You decide.

Hi @kefir090 ,
After community accusations of "benchmaxxing", we have evaluated your model on a few of our testing tools and it has unfortunately failed. Thus, we will be removing it model from the leaderboard.

Dan

Thankyou community for bringing this to my attention!

@Datdanboi25 Hi Dan,

Thanks for the note. Could you please share the specific evaluation logs, testing tools, or contamination metrics that triggered this decision?

It was probrably his private sets, like he has a Private Subset of some benchmarks thats not public so it cant be contaminated. But idk im just guesisng

Hi @Datdanboi25 and @Banaxi-Tech ,
I would like to clarify a major misunderstanding regarding the Nemotron-CLIMB paper (arXiv:2504.13161):

1. ClimbMix does NOT contain the text or tokens of the PIQA, ARC, or HellaSwag validation sets. There is zero data contamination or inclusion of benchmark questions/answers in the dataset.
2. How ClimbMix actually works: As detailed in Sections 2.2 and 3 of the paper, the validation splits of PIQA, ARC_E, and HellaSwag were used strictly as evaluation target metrics (calibration objective) for training a proxy LightGBM predictor. This predictor was used solely to search for the optimal mixing ratios (weights) among 20 web-text clusters (derived from Nemotron-CC and smollm-corpus).
3. The final 400B Nemotron-ClimbMix corpus consists purely of web data mixed at those optimal ratios. Evaluating proxy models on validation splits to tune hyperparameter ratios is standard data-mixture research, not dataset contamination.

Banaxi-Tech's claim that "the paper literally says validation sets are in the data" is a misinterpretation of how the proxy evaluation search was conducted.

Since there is no data leakage/contamination in Lumen-118M-Base, I kindly request you to re-evaluate the decision to remove the model from the leaderboard. Thanks!

Hello @kefir090 ,

Here are a few additional benchmark results that, while not exhaustive, show Lumen performing relatively poorly compared to GPT-X-125M across these unrelated benchmarks, despite being ahead on the benchmarks considered by the leaderboard.

To be clear, this isn't conclusive evidence of anything by itself. I just found the discrepancy interesting enough to share, especially given the relatively small training budget claimed for Lumen.

Hi @Datdanboi25 and @Banaxi-Tech ,
I would like to clarify a major misunderstanding regarding the Nemotron-CLIMB paper (arXiv:2504.13161):

1. ClimbMix does NOT contain the text or tokens of the PIQA, ARC, or HellaSwag validation sets. There is zero data contamination or inclusion of benchmark questions/answers in the dataset.
2. How ClimbMix actually works: As detailed in Sections 2.2 and 3 of the paper, the validation splits of PIQA, ARC_E, and HellaSwag were used strictly as evaluation target metrics (calibration objective) for training a proxy LightGBM predictor. This predictor was used solely to search for the optimal mixing ratios (weights) among 20 web-text clusters (derived from Nemotron-CC and smollm-corpus).
3. The final 400B Nemotron-ClimbMix corpus consists purely of web data mixed at those optimal ratios. Evaluating proxy models on validation splits to tune hyperparameter ratios is standard data-mixture research, not dataset contamination.

Banaxi-Tech's claim that "the paper literally says validation sets are in the data" is a misinterpretation of how the proxy evaluation search was conducted.

Since there is no data leakage/contamination in Lumen-118M-Base, I kindly request you to re-evaluate the decision to remove the model from the leaderboard. Thanks!

Yes they do include it.

Whats your AI generated response gonna be now?

chatgpt these people say my model is trained on benchmark data, thats true but can you pls make up a reason?

Then explain @User01110 findings and that BananaMind base bench is at 879, and my 3m models get 890 the one benchmark climbmix not included

Hello @kefir090 ,

Here are a few additional benchmark results that, while not exhaustive, show Lumen performing relatively poorly compared to GPT-X-125M across these unrelated benchmarks, despite being ahead on the benchmarks considered by the leaderboard.

Benchmark	Lumen-118M	GPT-X-125M
BLiMP	69.97%	82.29%
Winogrande	50.43%	52.17%
OpenBookQA	23.80%	31.20%
LAMBADA	0.50%	30.06%
WikiText BPB	2.24	0.96

To be clear, this isn't conclusive evidence of anything by itself. I just found the discrepancy interesting enough to share, especially given the relatively small training budget claimed for Lumen.

Hi @User01110 ,

Thanks for sharing! I just re-ran lm-eval directly on the Lumen-118M-Base checkpoint across those benchmarks to verify, and it looks like your harness setup had an issue with evaluation:

LAMBADA (OpenAI): Scores 27.25% (acc), not 0.50%
OpenBookQA: Scores 27.40% (acc_norm)
Winogrande: Scores 50.83% (acc)

The model evaluates completely as expected for a 118M model on 3.88B tokens. Appreciate you sharing the feedback!

@Banaxi-Tech , results on other benchmarks like Base Bench Elo and LAMBADA are good evidence that Lumen generalized poorly and was skewed toward the leaderboard tasks. However, CLIMB paper does not claim that the validation questions were added to its training set. Comparison of data mixtures (mixture a vs b), select the best one and evaluate on test sets based on validation scores. That is benchmark aware optimization, not evidence of direct pollution.

Hello @kefir090 ,

Here are a few additional benchmark results that, while not exhaustive, show Lumen performing relatively poorly compared to GPT-X-125M across these unrelated benchmarks, despite being ahead on the benchmarks considered by the leaderboard.

Benchmark	Lumen-118M	GPT-X-125M
BLiMP	69.97%	82.29%
Winogrande	50.43%	52.17%
OpenBookQA	23.80%	31.20%
LAMBADA	0.50%	30.06%
WikiText BPB	2.24	0.96

To be clear, this isn't conclusive evidence of anything by itself. I just found the discrepancy interesting enough to share, especially given the relatively small training budget claimed for Lumen.

Hi @User01110 ,

Thanks for sharing! I just re-ran lm-eval directly on the Lumen-118M-Base checkpoint across those benchmarks to verify, and it looks like your harness setup had an issue with evaluation:

LAMBADA (OpenAI): Scores 27.25% (acc), not 0.50%
OpenBookQA: Scores 27.40% (acc_norm)
Winogrande: Scores 50.83% (acc)

The model evaluates completely as expected for a 118M model on 3.88B tokens. Appreciate you sharing the feedback!

I reran it and i got the same results as User01110

@User01110 @Banaxi-Tech Just to follow up on those WikiText and LAMBADA numbers after checking with lm_eval,

For WikiText, the 2.14-2.24 number in the table is actually byte perplexity, not bits per byte. The real bits per byte score from lm_eval is 1.10, which is completely normal for a 118M model (GPT-2 gets around 1.10-1.20 as well).

For LAMBADA, lm_eval sets add_special_tokens=False by default. Since Lumen was pretrained with a BOS token at the start of every sequence, running it without BOS throws off the RoPE position alignment, which causes the LAMBADA accuracy to drop near zero. Passing add_bos_token=True in lm_eval fixes the alignment and brings the score back to the real 27.25%.

I already added add_bos_token=True

What exact lm_eval command did you run?

What exact lm_eval command did you run?

lemme check

lm_eval
--model hf
--model_args pretrained=kefir090/Lumen-118M-Base,add_bos_token=True,trust_remote_code=True
--tasks lambada_openai
--device cuda:0
--batch_size auto

This is what i get on lm eval

@Datdanboi25 Hi Dan,

Thanks for the note. Could you please share the specific evaluation logs, testing tools, or contamination metrics that triggered this decision?

Hi @kefir090 ,
To prevent people from either training on the benchmark, or creating a dataset specifically optimized for the benchmark, we maintain a series of hidden sets. When testing your model on these hidden sets, it demonstrated a >10% accuracy drop on all of them vs the public set, while typical gaps are in the range of +- 2-3%. Thus we have decided to delist the model.

Dan

Hi @Datdanboi25 We wanted to tell you that we are currently training a contamination classifier, ill give you more info soon.
if you want you could implement it into the Open SLM Leaderboard after we test its accuracy!
So when a model gets submitted it goes trough the classifier, then if it says no it gets added, if it says its contaminated we do manual review?
Thanks!

Yeah sounds good, would love to have it as a tool!