# Evaluation prompts (format and sources)

Each file is a **JSONL**: one JSON prompt per line.

## Line format

```json
{"id": "gsm8k-0001", "domain": "math", "text": "<full templated prompt>"}
```

- `id`: `<benchmark>-<num>` unique; it is the runner's resume key.
- `domain`: one of `math | code | chat | reasoning` (RQ1 domains).
- `text`: the already-templated prompt (no placeholders), ready for `llama-cli -p`.

## Sources (9 DeepSpec benchmarks — comparable with DSpark Table 1)

| Domain | Benchmarks |
|---|---|
| math | GSM8K, MATH-500, AIME25 |
| code | HumanEval, MBPP, LiveCodeBench |
| chat | MT-Bench, Alpaca, Arena-Hard-v2 |

## Subsamples

- **Fixed seed 42** for the whole study (reproducibility).
- ~100–200 prompts per domain (overnight GPU budget).
- `manifest.json` records the seed, the filter and the total per benchmark.

## Sets used in the campaigns

| File | Prompts | Campaign | Reference |
|---|---|---|---|
| `f1-sample.jsonl` | 1474 | final run (26 configs) | `scripts/run_final_chain.sh` |
| `old/acc-sample.jsonl` | 180 (stratified) | curves + k-sweep | `scripts/run_curves_chain.sh`, `scripts/run_ksweep_chain.sh` |

The sets are versioned exactly as used (do not edit: it would invalidate resume
and reproducibility). Raw dataset texts are NOT versioned
(`experiments/prompts/raw/` is in `.gitignore`).

## Rules

- Do not edit a subsample already used in `experiments/runs/` (it would
  invalidate resume and reproducibility). If it changes, create a new subsample
  with a version in the id.
- Raw dataset texts are NOT versioned here; only the JSONL subsamples (if their
  license allows it) and the manifest.