| # Evaluation prompts (format and sources) |
|
|
| Each file is a **JSONL**: one JSON prompt per line. |
|
|
| ## Line format |
|
|
| ```json |
| {"id": "gsm8k-0001", "domain": "math", "text": "<full templated prompt>"} |
| ``` |
|
|
| - `id`: `<benchmark>-<num>` unique; it is the runner's resume key. |
| - `domain`: one of `math | code | chat | reasoning` (RQ1 domains). |
| - `text`: the already-templated prompt (no placeholders), ready for `llama-cli -p`. |
|
|
| ## Sources (9 DeepSpec benchmarks — comparable with DSpark Table 1) |
|
|
| | Domain | Benchmarks | |
| |---|---| |
| | math | GSM8K, MATH-500, AIME25 | |
| | code | HumanEval, MBPP, LiveCodeBench | |
| | chat | MT-Bench, Alpaca, Arena-Hard-v2 | |
|
|
| ## Subsamples |
|
|
| - **Fixed seed 42** for the whole study (reproducibility). |
| - ~100–200 prompts per domain (overnight GPU budget). |
| - `manifest.json` records the seed, the filter and the total per benchmark. |
|
|
| ## Sets used in the campaigns |
|
|
| | File | Prompts | Campaign | Reference | |
| |---|---|---|---| |
| | `f1-sample.jsonl` | 1474 | final run (26 configs) | `scripts/run_final_chain.sh` | |
| | `old/acc-sample.jsonl` | 180 (stratified) | curves + k-sweep | `scripts/run_curves_chain.sh`, `scripts/run_ksweep_chain.sh` | |
|
|
| The sets are versioned exactly as used (do not edit: it would invalidate resume |
| and reproducibility). Raw dataset texts are NOT versioned |
| (`experiments/prompts/raw/` is in `.gitignore`). |
|
|
| ## Rules |
|
|
| - Do not edit a subsample already used in `experiments/runs/` (it would |
| invalidate resume and reproducibility). If it changes, create a new subsample |
| with a version in the id. |
| - Raw dataset texts are NOT versioned here; only the JSONL subsamples (if their |
| license allows it) and the manifest. |
|
|
