Byrne-100M-Ultra-MC
New Model: Byrne-100M-Ultra-MC - https://huggingface.co/Quazim0t0/Byrne-100M-Ultra-MC
The one link includes the base, sft and dpo.
| Metric | base-62k | sft-7100 | dpo-3200 | chance |
|---|---|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.308 | 2.383 | 2.385 | — |
| BLiMP acc ↑ | 0.811 | 0.780 | 0.779 | 0.50 |
| ARC-Easy acc | 0.410 | 0.395 | 0.400 | 0.25 |
| ARC-Easy acc_norm | 0.390 | 0.400 | 0.400 | 0.25 |
| ARC-Challenge acc | 0.205 | 0.265 | 0.255 | 0.25 |
| ARC-Challenge acc_norm | 0.270 | 0.260 | 0.260 | 0.25 |
| HellaSwag acc | 0.370 | 0.375 | 0.370 | 0.25 |
| HellaSwag acc_norm | 0.435 | 0.420 | 0.415 | 0.25 |
| Winogrande acc | 0.515 | 0.500 | 0.505 | 0.50 |
| PIQA acc | 0.565 | 0.590 | 0.590 | 0.50 |
| PIQA acc_norm | 0.555 | 0.575 | 0.575 | 0.50 |
| OpenBookQA acc | 0.105 | 0.100 | 0.100 | 0.25 |
| OpenBookQA acc_norm | 0.290 | 0.285 | 0.295 | 0.25 |
| BoolQ acc | 0.355 | 0.430 | 0.425 | 0.50 |
| ArithMark-3.0 acc_norm | 0.354 | 0.380 | 0.378 | 0.25 |
Oh man I totally forgot to upload. Hold on, I'll be uploading now.
@Datdanboi25 ### Benchmarks (full)
Same harness, no 200-example cap. WikiText-2 byte_ppl and BLiMP are the
same numbers as above (those were already full). MC tasks and ArithMark
are the whole set.
n: ARC-Easy 2,376, ARC-Challenge 1,172, HellaSwag 10,042, Winogrande
1,267, PIQA 1,838, OpenBookQA 500, BoolQ 3,270, ArithMark 1,000.
| Metric | base-62k | sft-7100 | dpo-3200 | chance |
|---|---|---|---|---|
| WikiText-2 byte_ppl ↓ | 2.308 | 2.383 | 2.385 | — |
| BLiMP acc ↑ | 0.811 | 0.780 | 0.779 | 0.50 |
| ARC-Easy acc | 0.429 | 0.411 | 0.411 | 0.25 |
| ARC-Easy acc_norm | 0.394 | 0.388 | 0.388 | 0.25 |
| ARC-Challenge acc | 0.190 | 0.208 | 0.206 | 0.25 |
| ARC-Challenge acc_norm | 0.220 | 0.227 | 0.230 | 0.25 |
| HellaSwag acc | 0.278 | 0.281 | 0.281 | 0.25 |
| HellaSwag acc_norm | 0.293 | 0.291 | 0.290 | 0.25 |
| Winogrande acc | 0.515 | 0.521 | 0.518 | 0.50 |
| PIQA acc | 0.583 | 0.584 | 0.583 | 0.50 |
| PIQA acc_norm | 0.583 | 0.584 | 0.583 | 0.50 |
| OpenBookQA acc | 0.118 | 0.120 | 0.120 | 0.25 |
| OpenBookQA acc_norm | 0.242 | 0.240 | 0.244 | 0.25 |
| BoolQ acc | 0.381 | 0.418 | 0.416 | 0.50 |
| ArithMark-3.0 acc_norm | 0.358 | 0.372 | 0.373 | 0.25 |
HellaSwag acc_norm drops from the capped 0.435 / 0.420 / 0.415 to ~0.29
once you take all 10k items. ARC-Challenge SFT bump shrinks too (+0.06
capped, +0.018 full). BoolQ still moves (0.381 → 0.418). DPO and SFT
stay within noise. OpenBookQA acc is still under chance.
ok great added the base model!
@Datdanboi25 Hey Dan, I just noticed it says 114x4 but its 114x3 if the three represents the loops and I'm confused how it scored 32% on your run, but I benchmarked it twice due to last time when you mentioned that my Escarda model scored high, so I made sure to benchmark twice now. What was the difference between your benchmark and my own?
Also, I am not disagreeing with you. I just want to know for any future models I submit later on so they will be done correctly.
Hi @Quazim0t0 ,
You were right, finally got around to having a look though the adapter wrapper and there was a slight bug in the arithmark scoring, my apologies.
@Datdanboi25 Hey Dan, could you clarify if I am supposed to be benchmarking my models with all 10k HellaSwag lol? It takes so long to benchmark every time..
The scores posted above were from the full benchmark sets, not the LM-Harness: ARC-Easy 2,376, ARC-Challenge 1,172, HellaSwag 10,042, Winogrande 1,267, PIQA 1,838, OpenBookQA 500, BoolQ 3,270, ArithMark 1,000.
Yeah, all 10k hellaswag Questions haha, takes a while
The tokenizer I'm using with my model causes issues with the LM-Eval Harness, so I had to try and recreate it, and I didn't know if my benchmarks were set to the right configuration. So, I ended up doing a capped 200 and the full. Thank you for clarifying.
I'm just going to leave it running when I sleep from now on, I had no idea it was doing 10k every time lol.
Oh no worries, I checked all ur scores on my system and looks all good!