Evaluation prompts (format and sources)

Each file is a JSONL: one JSON prompt per line.

Line format

{"id": "gsm8k-0001", "domain": "math", "text": "<full templated prompt>"}

id: <benchmark>-<num> unique; it is the runner's resume key.
domain: one of math | code | chat | reasoning (RQ1 domains).
text: the already-templated prompt (no placeholders), ready for llama-cli -p.

Sources (9 DeepSpec benchmarks — comparable with DSpark Table 1)

Domain	Benchmarks
math	GSM8K, MATH-500, AIME25
code	HumanEval, MBPP, LiveCodeBench
chat	MT-Bench, Alpaca, Arena-Hard-v2

Subsamples

Fixed seed 42 for the whole study (reproducibility).
~100–200 prompts per domain (overnight GPU budget).
manifest.json records the seed, the filter and the total per benchmark.

Sets used in the campaigns

File	Prompts	Campaign	Reference
`f1-sample.jsonl`	1474	final run (26 configs)	`scripts/run_final_chain.sh`
`old/acc-sample.jsonl`	180 (stratified)	curves + k-sweep	`scripts/run_curves_chain.sh`, `scripts/run_ksweep_chain.sh`

The sets are versioned exactly as used (do not edit: it would invalidate resume and reproducibility). Raw dataset texts are NOT versioned (experiments/prompts/raw/ is in .gitignore).

Rules

Do not edit a subsample already used in experiments/runs/ (it would invalidate resume and reproducibility). If it changes, create a new subsample with a version in the id.
Raw dataset texts are NOT versioned here; only the JSONL subsamples (if their license allows it) and the manifest.