README.md

metadata

base_model:
  - SicariusSicariiStuff/Impish_Magic_24B

vllm (pretrained=/root/autodl-tmp/Impish_Magic_24B,add_bos_token=true,max_model_len=3096,dtype=bfloat16,trust_remote_code=true,gpu_memory_utilization=0.8), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.908	±	0.0183
		strict-match	5	exact_match	↑	0.904	±	0.0187

vllm (pretrained=/root/autodl-tmp/Impish_Magic_24B,add_bos_token=true,max_model_len=3048,dtype=bfloat16,trust_remote_code=true,gpu_memory_utilization=0.9), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.7545	±	0.0034
- humanities	2	none		acc	↑	0.6782	±	0.0064
- other	2	none		acc	↑	0.8169	±	0.0067
- social sciences	2	none		acc	↑	0.8469	±	0.0064
- stem	2	none		acc	↑	0.7168	±	0.0077

vllm (pretrained=/root/autodl-tmp/90-512-2048-9999999,add_bos_token=true,max_model_len=3096,dtype=bfloat16,trust_remote_code=true,gpu_memory_utilization=0.8), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.908	±	0.0183
		strict-match	5	exact_match	↑	0.908	±	0.0183

vllm (pretrained=/root/autodl-tmp/90-512-2048-9999999,add_bos_token=true,max_model_len=3048,dtype=bfloat16,trust_remote_code=true,gpu_memory_utilization=0.9), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.7527	±	0.0034
- humanities	2	none		acc	↑	0.6810	±	0.0064
- other	2	none		acc	↑	0.8072	±	0.0068
- social sciences	2	none		acc	↑	0.8463	±	0.0064
- stem	2	none		acc	↑	0.7146	±	0.0077