Problem files
scripts/prepare_data.py writes the five problem files that every lab reads.
| File | What it holds |
|---|---|
math500.jsonl |
The MATH-500 test set (500 problems), one of the paper's three benchmarks |
gsm8k.jsonl |
The GSM8K main test set (1,319 problems), a benchmark |
ifeval.jsonl |
IFEval (541 prompts), a benchmark graded by its official checkers |
dev256.jsonl |
128 GSM8K-train and 128 MATH-train practice problems the paper's agent never used. We measure every edit here. |
build128.jsonl |
64 + 64 more unused practice problems. The Lab 8 capstone builds its own edit from these. |
The copy of this project on Hugging Face leaves the .jsonl files out. Three of them are benchmark test sets, and test sets posted online can end up in the training data of future models. The script rebuilds all five from public sources.
To rebuild them, run this from the project root on a machine with internet access and the datasets package (scripts/setup_vm.sh installs it on the GPU machines):
git clone https://github.com/RikiyaT/lm_understand-logs external/lm_understand-logs
git -C external/lm_understand-logs checkout 4d09ba6d # the commit we used
python scripts/prepare_data.py --exp8 external/lm_understand-logs/exp8_math --out data