| hf (pretrained=/mnt/jfzn/msj/train_exp/gated_deltaproduct,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 4 | |
| | Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr| | |
| |----------------|------:|------|-----:|---------------|---|------:|---|------| | |
| |arc_challenge | 1|none | 0|acc |↑ | 0.2739|± |0.0130| | |
| | | |none | 0|acc_norm |↑ | 0.2995|± |0.0134| | |
| |arc_easy | 1|none | 0|acc |↑ | 0.6069|± |0.0100| | |
| | | |none | 0|acc_norm |↑ | 0.5492|± |0.0102| | |
| |hellaswag | 1|none | 0|acc |↑ | 0.4292|± |0.0049| | |
| | | |none | 0|acc_norm |↑ | 0.5534|± |0.0050| | |
| |lambada_standard| 1|none | 0|acc |↑ | 0.4710|± |0.0070| | |
| | | |none | 0|perplexity |↓ |12.1930|± |0.3457| | |
| |piqa | 1|none | 0|acc |↑ | 0.7291|± |0.0104| | |
| | | |none | 0|acc_norm |↑ | 0.7225|± |0.0104| | |
| |wikitext | 2|none | 0|bits_per_byte |↓ | 0.7348|± | N/A| | |
| | | |none | 0|byte_perplexity|↓ | 1.6642|± | N/A| | |
| | | |none | 0|word_perplexity|↓ |15.2375|± | N/A| | |
| |winogrande | 1|none | 0|acc |↑ | 0.5927|± |0.0138| |