| --- |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - en |
| --- |
| |
| Smallstral, layer pruned Mistral |
| | Task | Metric | Mistral-7B-v0.1 Value | Smallstral Value | |
| |----------------------------------------------|----------|----------------------:|-----------------:| |
| | hendrycksTestRu-high_school_mathematics | acc | 0.3037 | 0.2852 | |
| | hendrycksTestRu-high_school_microeconomics | acc | 0.3950 | 0.3319 | |
| | hendrycksTestRu-high_school_physics | acc | 0.3113 | 0.2649 | |
| | hendrycksTestRu-high_school_psychology | acc | 0.5193 | 0.1945 | |
| | hendrycksTestRu-high_school_statistics | acc | 0.3241 | 0.4028 | |
| | hendrycksTestRu-high_school_us_history | acc | 0.5392 | 0.2451 | |
| | hendrycksTestRu-high_school_world_history | acc | 0.5781 | 0.2321 | |
| | hendrycksTestRu-human_aging | acc | 0.4843 | 0.3184 | |
| | parus | acc | 0.5000 | 0.4875 | |
| | rcb | acc | 0.3653 | 0.3288 | |
| | rcb | f1_macro | 0.3313 | 0.1691 | |
| | rwsd | acc | 0.4917 | 0.4901 | |
