OOS test set (TS_Caption_test) — faithful rebuild from raw (used datasets only)
Reproduces the OOS request files we actually use, byte-for-byte, from the raw
TS_Caption_test. bash build_oos_test.sh writes exactly these 5:
| file | what | md5 |
|---|---|---|
threesplit_501ts_promptV2.jsonl |
metric-QA (zh) — champion/joint headline | 154306d835bb... |
caption_501ts_3split.jsonl |
caption (zh, canonical) | 820fe34d2997... |
caption_3split_501ts_para1.jsonl |
caption zh paraphrase #1 | 11dce0be3dd3... |
caption_3split_501ts_para2.jsonl |
caption zh paraphrase #2 | ced4dfcf28b7... |
caption_3split_501ts_en.jsonl |
caption en mirror | b4fa39d517e6... |
Pipeline
_build/build_oos_v2.py--seed 42 (patched to emit only the 2 used files; the mvonly_* / threesplit_NEWv1 / caption_500ts_mvonly variants are commented out,# [release: unused, not generated]). Reads raw univar/bivar/multivar; QA prompt viats_align.data.sample_stage2_prompt._build/build_caption_variants.py(patched to 3split-only) — swaps the caption prompt with hardcoded para1/para2/en templates (deterministic, no external LLM), all other fields bit-identical.
Deliberately NOT built here
- mvonly / NEWv1 QA + caption_500ts_mvonly — variants we don't use (not generated).
threesplit_..._promptV2_EN.jsonl(QA English) — needs an LLM translation (build_en_qa.py, GLM/Anthropic); can't be byte-rebuilt deterministically. It's an optional (commented) step in the driver:export ANTHROPIC_AUTH_TOKEN=... ANTHROPIC_BASE_URL=... EXTRACTOR_MODEL=glm-5.1then uncomment. Or copy the existing canonical file (761f965c).
Files
build_oos_test.sh— driver (build + md5 check)_build/{build_oos_v2.py, build_caption_variants.py, ts_align/{data,qa_templates,__init__}.py}- raw source:
../Benchmarks/TS_Caption_test/{univar,bivar,multivar}/
Note
QA (154306d835) is reproducible only against the current _build/ts_align/data.py
(599bb34a) prompt pool — pin it if the QA md5 ever drifts. Caption + variants have no
prompt-pool dependency.