data/math/README.md

# MATH data files (graphrpo)

Honest naming of the MATH eval/train parquets. See
`experiments/verl/notes/reproduction_plan.md` for the full analysis.

| file | what it actually is | rows | level mix (L1/L2/L3/L4/L5) | use |
|------|--------------------|------|----------------------------|-----|
| `train.parquet` | MATH-lighteval **train** split | 7500 | — | training |
| `test.parquet` | MATH-lighteval **test** split (full) | 5000 | — | source of the slice below |
| `test500.parquet` | **`test.parquet[:500]` — a PREFIX SLICE, *not* MATH-500** | 500 | 64/82/114/123/117 | legacy eval; still read by in-flight jobs |
| `math_test_first500.parquet` | byte-identical copy of `test500.parquet` (honest name) | 500 | 64/82/114/123/117 | prefer this name going forward for the slice |
| `math500_official.parquet` | **the TRUE MATH-500** (Lightman et al. / PRM800K subset, `HuggingFaceH4/MATH-500`) | 497* | 43/90/105/128/134 (of the canonical 500) | standard benchmark eval |

\* The canonical MATH-500 has **500** problems with mix L1:43 L2:90 L3:105 L4:128 L5:134.
3 problems tokenize to >512 tokens under the Qwen2.5-1.5B-Instruct chat template and are
dropped by verl's `filter_overlong_prompts` (mirrored at build time), leaving **497** rows.
Emitted token lengths: min 52, max 504, mean 111.

## Why this matters
`test500.parquet` is **not** the published MATH-500 benchmark. Its level mix is easier
(more Level-1, fewer Level-5) than canonical MATH-500, so any "MATH-500" number reported
on it is on a non-standard, non-comparable scale. Reproductions and paper numbers must use
`math500_official.parquet`.

## Do not delete
`test500.parquet` is left in place because in-flight jobs read it. New work should use
`math500_official.parquet` (real benchmark) or `math_test_first500.parquet` (the slice,
honestly named).

Built by `experiments/verl/repro/prep_math500.py` (source `HuggingFaceH4/MATH-500`).