Benchmarks?

#4
by LH-Tech-AI - opened

Hi, could you run benchmarks like arc easy, arc challenge, hellaswag and piqa on the model please to evaluate it.
Thank you. Then please share acc_norm.
β€οΈπŸ€—

I love the project πŸ”₯πŸ”₯πŸ”₯

At the current state of training, running benchmarks would be absolutely desastrous 😭😭... Yhe model was trained mainly on english for the first 25B tokens, and now it bearly learns programming/math/instruction following... Benchmarking it yet would not be very useful. I will run benchmarks when the model will have been trained on 100B tokens though.

Okay that's fine. Thanks for the answer though. But benchmarks @ ~38B tokens would be interesting though.

the 38B tokens are not real 😭 we lose ~x3 from cross checks and we lose a bit of convergence forcing the rounds to be 4 hours long on server (1 huge SGD step < 100 smaller backprop training steps)

ah okay. got it.

LH-Tech-AI changed discussion status to closed
Vxtzq changed discussion status to open

Wait, benchmarks might still come before that! Someone on the project discord suggested that we implement benchmarks on the site, with a simple github file named benchmarks.txt (that anyone can contribute to), which is looked up by your browser and then parsed onto the site :)

I hope you find this good enough!

Yeah that's nice. Tell me when it's there! πŸ€—

Well early stage benchmarks show that
image
It's somehow good on QA even though in my experiments it wasn't that good xD

nice thanks

I also put better represented benchmarks on the GitHub readme πŸ€—

Sign up or log in to comment