File size: 1,561 Bytes
6bd3c24 6da8687 6bd3c24 6da8687 6bd3c24 2bd5681 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
---
library_name: transformers
license: mit
datasets:
- kjj0/fineweb100B-gpt2
---
trained on 12,312,444,928 tokens from the [kjj0/fineweb100B-gpt2](https://huggingface.co/datasets/kjj0/fineweb100B-gpt2) dataset
```
$ lm_eval --model hf \
--model_args pretrained=michaelbzhu/test-7.6B-base,trust_remote_code=True \
--tasks mmlu_college_medicine,hellaswag,lambada_openai,arc_easy,winogrande,arc_challenge,openbookqa \
--device cuda:0 \
--batch_size 16
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr|
|----------------|------:|------|-----:|----------|---|------:|---|-----:|
|arc_challenge | 1|none | 0|acc |↑ | 0.2295|± |0.0123|
| | |none | 0|acc_norm |↑ | 0.2628|± |0.0129|
|arc_easy | 1|none | 0|acc |↑ | 0.5358|± |0.0102|
| | |none | 0|acc_norm |↑ | 0.4663|± |0.0102|
|hellaswag | 1|none | 0|acc |↑ | 0.3788|± |0.0048|
| | |none | 0|acc_norm |↑ | 0.4801|± |0.0050|
|lambada_openai | 1|none | 0|acc |↑ | 0.4527|± |0.0069|
| | |none | 0|perplexity|↓ |14.3601|± |0.4468|
|college_medicine| 1|none | 0|acc |↑ | 0.2254|± |0.0319|
|openbookqa | 1|none | 0|acc |↑ | 0.1920|± |0.0176|
| | |none | 0|acc_norm |↑ | 0.3020|± |0.0206|
|winogrande | 1|none | 0|acc |↑ | 0.5107|± |0.0140|
```
|