File size: 1,653 Bytes
d4c2896 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | # Evaluation prompts (format and sources)
Each file is a **JSONL**: one JSON prompt per line.
## Line format
```json
{"id": "gsm8k-0001", "domain": "math", "text": "<full templated prompt>"}
```
- `id`: `<benchmark>-<num>` unique; it is the runner's resume key.
- `domain`: one of `math | code | chat | reasoning` (RQ1 domains).
- `text`: the already-templated prompt (no placeholders), ready for `llama-cli -p`.
## Sources (9 DeepSpec benchmarks — comparable with DSpark Table 1)
| Domain | Benchmarks |
|---|---|
| math | GSM8K, MATH-500, AIME25 |
| code | HumanEval, MBPP, LiveCodeBench |
| chat | MT-Bench, Alpaca, Arena-Hard-v2 |
## Subsamples
- **Fixed seed 42** for the whole study (reproducibility).
- ~100–200 prompts per domain (overnight GPU budget).
- `manifest.json` records the seed, the filter and the total per benchmark.
## Sets used in the campaigns
| File | Prompts | Campaign | Reference |
|---|---|---|---|
| `f1-sample.jsonl` | 1474 | final run (26 configs) | `scripts/run_final_chain.sh` |
| `old/acc-sample.jsonl` | 180 (stratified) | curves + k-sweep | `scripts/run_curves_chain.sh`, `scripts/run_ksweep_chain.sh` |
The sets are versioned exactly as used (do not edit: it would invalidate resume
and reproducibility). Raw dataset texts are NOT versioned
(`experiments/prompts/raw/` is in `.gitignore`).
## Rules
- Do not edit a subsample already used in `experiments/runs/` (it would
invalidate resume and reproducibility). If it changes, create a new subsample
with a version in the id.
- Raw dataset texts are NOT versioned here; only the JSONL subsamples (if their
license allows it) and the manifest.
|