Byrne-100M-Ultra-MC
New Model: Byrne-100M-Ultra-MC - https://huggingface.co/Quazim0t0/Byrne-100M-Ultra-MC
The one link includes the base, sft and dpo.
| Metric | base-62k | sft-7100 | dpo-3200 | chance |
|---|---|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.308 | 2.383 | 2.385 | — |
| BLiMP acc ↑ | 0.811 | 0.780 | 0.779 | 0.50 |
| ARC-Easy acc | 0.410 | 0.395 | 0.400 | 0.25 |
| ARC-Easy acc_norm | 0.390 | 0.400 | 0.400 | 0.25 |
| ARC-Challenge acc | 0.205 | 0.265 | 0.255 | 0.25 |
| ARC-Challenge acc_norm | 0.270 | 0.260 | 0.260 | 0.25 |
| HellaSwag acc | 0.370 | 0.375 | 0.370 | 0.25 |
| HellaSwag acc_norm | 0.435 | 0.420 | 0.415 | 0.25 |
| Winogrande acc | 0.515 | 0.500 | 0.505 | 0.50 |
| PIQA acc | 0.565 | 0.590 | 0.590 | 0.50 |
| PIQA acc_norm | 0.555 | 0.575 | 0.575 | 0.50 |
| OpenBookQA acc | 0.105 | 0.100 | 0.100 | 0.25 |
| OpenBookQA acc_norm | 0.290 | 0.285 | 0.295 | 0.25 |
| BoolQ acc | 0.355 | 0.430 | 0.425 | 0.50 |
| ArithMark-3.0 acc_norm | 0.354 | 0.380 | 0.378 | 0.25 |
Oh man I totally forgot to upload. Hold on, I'll be uploading now.
@Datdanboi25 ### Benchmarks (full)
Same harness, no 200-example cap. WikiText-2 byte_ppl and BLiMP are the
same numbers as above (those were already full). MC tasks and ArithMark
are the whole set.
n: ARC-Easy 2,376, ARC-Challenge 1,172, HellaSwag 10,042, Winogrande
1,267, PIQA 1,838, OpenBookQA 500, BoolQ 3,270, ArithMark 1,000.
| Metric | base-62k | sft-7100 | dpo-3200 | chance |
|---|---|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.308 | 2.383 | 2.385 | — |
| BLiMP acc ↑ | 0.811 | 0.780 | 0.779 | 0.50 |
| ARC-Easy acc | 0.429 | 0.411 | 0.411 | 0.25 |
| ARC-Easy acc_norm | 0.394 | 0.388 | 0.388 | 0.25 |
| ARC-Challenge acc | 0.190 | 0.208 | 0.206 | 0.25 |
| ARC-Challenge acc_norm | 0.220 | 0.227 | 0.230 | 0.25 |
| HellaSwag acc | 0.278 | 0.281 | 0.281 | 0.25 |
| HellaSwag acc_norm | 0.293 | 0.291 | 0.290 | 0.25 |
| Winogrande acc | 0.515 | 0.521 | 0.518 | 0.50 |
| PIQA acc | 0.583 | 0.584 | 0.583 | 0.50 |
| PIQA acc_norm | 0.583 | 0.584 | 0.583 | 0.50 |
| OpenBookQA acc | 0.118 | 0.120 | 0.120 | 0.25 |
| OpenBookQA acc_norm | 0.242 | 0.240 | 0.244 | 0.25 |
| BoolQ acc | 0.381 | 0.418 | 0.416 | 0.50 |
| ArithMark-3.0 acc_norm | 0.358 | 0.372 | 0.373 | 0.25 |
HellaSwag acc_norm drops from the capped 0.435 / 0.420 / 0.415 to ~0.29
once you take all 10k items. ARC-Challenge SFT bump shrinks too (+0.06
capped, +0.018 full). BoolQ still moves (0.381 → 0.418). DPO and SFT
stay within noise. OpenBookQA acc is still under chance.
ok great added the base model!
@Datdanboi25 Hey Dan, I just noticed it says 114x4 but its 114x3 if the three represents the loops and I'm confused how it scored 32% on your run, but I benchmarked it twice due to last time when you mentioned that my Escarda model scored high, so I made sure to benchmark twice now. What was the difference between your benchmark and my own?
Also, I am not disagreeing with you. I just want to know for any future models I submit later on so they will be done correctly.
Hi @Quazim0t0 ,
You were right, finally got around to having a look though the adapter wrapper and there was a slight bug in the arithmark scoring, my apologies.