Diffulex JSON benchmarks
Bundled tasks (lm-eval names, use with --include_path diffulex_bench/tasks — the bench adds this by default):
| JSON | Task name |
|---|---|
GSM8K.json |
gsm8k_diffulex |
MATH500.json |
math500_diffulex, math500_diffulex_4shot, math500_diffulex_n32 |
HumanEval.json |
humaneval_diffulex |
MBPP.json |
mbpp_diffulex |
HumanEval_dmax.json |
humaneval_dmax_reference_chat |
MBPP_dmax.json |
mbpp_dmax_reference_chat |
The _dmax JSON files are dedicated copies for DMax-specific code tasks so their prompt/output
contract can evolve independently from the generic diffulex / SDAR-style tasks.