Benchmarks?
Hi, could you run benchmarks like arc easy, arc challenge, hellaswag and piqa on the model please to evaluate it.
Thank you. Then please share acc_norm.
β€οΈπ€
I love the project π₯π₯π₯
At the current state of training, running benchmarks would be absolutely desastrous ππ... Yhe model was trained mainly on english for the first 25B tokens, and now it bearly learns programming/math/instruction following... Benchmarking it yet would not be very useful. I will run benchmarks when the model will have been trained on 100B tokens though.
Okay that's fine. Thanks for the answer though. But benchmarks @ ~38B tokens would be interesting though.
the 38B tokens are not real π we lose ~x3 from cross checks and we lose a bit of convergence forcing the rounds to be 4 hours long on server (1 huge SGD step < 100 smaller backprop training steps)
ah okay. got it.
Wait, benchmarks might still come before that! Someone on the project discord suggested that we implement benchmarks on the site, with a simple github file named benchmarks.txt (that anyone can contribute to), which is looked up by your browser and then parsed onto the site :)
I hope you find this good enough!
Yeah that's nice. Tell me when it's there! π€
nice thanks
I also put better represented benchmarks on the GitHub readme π€
