TobiasLogic

/

Museko-125M

like 8

UPDATE: It was a error in my script, i fixed it and now it got 7%.

Thanks for flagging this, and sorry for the trouble.

I looked into it and the weights are fine. The model scores normally on the likelihood benchmarks (HellaSwag, ARC, PIQA, ArithMark), which means the forward pass and the probabilities are correct. The problem was the default decoding.

Museko is a base model and it was shipping with plain greedy decoding as the default. Small base models tend to fall into repetition loops under greedy, so a prompt like "Once upon a time," would collapse into "I was sitting there, and I was sitting there, and I was sitting there". Arithmetic still looked fine because those answers are a single deterministic string, but open ended prose looped, and any benchmark that does exact match on that looped text scores 0.

I just pushed a fix. The generation_config now sets a repetition penalty of 1.3 and no_repeat_ngram_size of 3 by default, so a plain model.generate() produces coherent text out of the box while staying deterministic. Example after the fix:

"Once upon a time, I was in the middle of my day and had to sit down at the table. I sat there for 10 minutes until I got up from my chair and walked out into the room."

Two things worth knowing. It is a base model, so use it for text completion rather than instructions or question answering. And if you want more varied output you can turn on sampling with do_sample=True, temperature=0.8, top_p=0.95. Please pull the latest revision and try again, and let me know if your benchmark looks better now.

actually, i think i made a mistake too with greedy, il upload a fix rq

Thanks alot for the fix

btw how many tokens have you trained it on

The benchmark is now 18%, that's good my 50m is 29% so its not have the issue anymore now

btw how many tokens have you trained it on

about 15B tokens, but its not trained to 100%, this model is at 61% trained when i was just getting tired of it and pulled the plug

The benchmark is now 18%, that's good my 50m is 29% so its not have the issue anymore now

can you tell me which benchmark are you using or is it private?

The benchmark is now 18%, that's good my 50m is 29% so its not have the issue anymore now

can you tell me which benchmark are you using or is it private?

Its my own Benchmark, private we have one public for instruct, but this one is for base and not public. We are going to release a Public Version soon though.

The benchmark is now 18%, that's good my 50m is 29% so its not have the issue anymore now

is that the https://huggingface.co/BananaMind/BananaMind-2-Medium ?

yea

Also know, for the https://huggingface.co/BananaMind/BananaMind-2-Medium-Chat that BananaMind Instruction Bench (Internal)
Is NOT this benchmark. Its another internal one not published yet

The benchmark is now 18%, that's good my 50m is 29% so its not have the issue anymore now

yeah that tracks. your 50m is a better free-text generator than museko for its size, ill give you that. museko is pretty skewed toward the multiple-choice benchmarks and arithmetic, its not great at generating on its own. i actually tested it and it only gets ~16% on arithmetic when it has to write the answer out vs ~75% on the multiple choice version, so on a base free-gen benchmark bananamind beating it doesnt surprise me. different strengths. nice work on the 50m tbh

I really like your model architecture. It seems like it could become very good when the training data gets scaled up from just fineweb edu to also adding cosmopedia v2 and DCLM!
The only thing i dont like that much is the arithmetic data because it kind of inflates the leaderboard score.

I really like your model architecture. It seems like it could become very good when the training data gets scaled up from just fineweb edu to also adding cosmopedia v2 and DCLM!
The only thing i dont like that much is the arithmetic data because it kind of inflates the leaderboard score.

Yeah this model was just design to get the top 1 in anyway possible, they should implement a system to discourage people from doing that, and this model isn't trained to it's potential (61% trained), and still a base model

I really like your model architecture. It seems like it could become very good when the training data gets scaled up from just fineweb edu to also adding cosmopedia v2 and DCLM!
The only thing i dont like that much is the arithmetic data because it kind of inflates the leaderboard score.

Yeah this model was just design to get the top 1 in anyway possible, they should implement a system to discourage people from doing that, and this model isn't trained to it's potential (61% trained), and still a base model

Yes Finally someone Agrees! We're already sending PR's to them to fix this like https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard/discussions/56

I really like your model architecture. It seems like it could become very good when the training data gets scaled up from just fineweb edu to also adding cosmopedia v2 and DCLM!
The only thing i dont like that much is the arithmetic data because it kind of inflates the leaderboard score.

Yeah this model was just design to get the top 1 in anyway possible, they should implement a system to discourage people from doing that, and this model isn't trained to it's potential (61% trained), and still a base model

Yes Finally someone Agrees! We're already sending PR's to them to fix this like https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard/discussions/56

Yeah, I think that's for the best.