File size: 678 Bytes
31dc8dc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # Diffulex JSON benchmarks
Bundled tasks (lm-eval names, use with `--include_path diffulex_bench/tasks` — the bench adds this by default):
| JSON | Task name |
|------|-----------|
| `GSM8K.json` | `gsm8k_diffulex` |
| `MATH500.json` | `math500_diffulex`, `math500_diffulex_4shot`, `math500_diffulex_n32` |
| `HumanEval.json` | `humaneval_diffulex` |
| `MBPP.json` | `mbpp_diffulex` |
| `HumanEval_dmax.json` | `humaneval_dmax_reference_chat` |
| `MBPP_dmax.json` | `mbpp_dmax_reference_chat` |
The `_dmax` JSON files are dedicated copies for DMax-specific code tasks so their prompt/output
contract can evolve independently from the generic `diffulex` / SDAR-style tasks.
|