| # OOS test set (TS_Caption_test) — faithful rebuild from raw (used datasets only) |
|
|
| Reproduces the OOS request files **we actually use**, byte-for-byte, from the raw |
| `TS_Caption_test`. `bash build_oos_test.sh` writes exactly these 5: |
|
|
| | file | what | md5 | |
| |---|---|---| |
| | `threesplit_501ts_promptV2.jsonl` | metric-QA (zh) — champion/joint headline | `154306d835bb...` | |
| | `caption_501ts_3split.jsonl` | caption (zh, canonical) | `820fe34d2997...` | |
| | `caption_3split_501ts_para1.jsonl`| caption zh paraphrase #1 | `11dce0be3dd3...` | |
| | `caption_3split_501ts_para2.jsonl`| caption zh paraphrase #2 | `ced4dfcf28b7...` | |
| | `caption_3split_501ts_en.jsonl` | caption en mirror | `b4fa39d517e6...` | |
|
|
| ## Pipeline |
| 1. **`_build/build_oos_v2.py` --seed 42** (patched to emit **only** the 2 used files; |
| the mvonly_* / threesplit_NEWv1 / caption_500ts_mvonly variants are commented out, |
| `# [release: unused, not generated]`). Reads raw univar/bivar/multivar; QA prompt |
| via `ts_align.data.sample_stage2_prompt`. |
| 2. **`_build/build_caption_variants.py`** (patched to 3split-only) — swaps the caption |
| prompt with hardcoded para1/para2/en templates (deterministic, **no external LLM**), |
| all other fields bit-identical. |
|
|
| ## Deliberately NOT built here |
| - **mvonly / NEWv1 QA + caption_500ts_mvonly** — variants we don't use (not generated). |
| - **`threesplit_..._promptV2_EN.jsonl` (QA English)** — needs an LLM translation |
| (`build_en_qa.py`, GLM/Anthropic); can't be byte-rebuilt deterministically. It's an |
| optional (commented) step in the driver: `export ANTHROPIC_AUTH_TOKEN=... ANTHROPIC_BASE_URL=... |
| EXTRACTOR_MODEL=glm-5.1` then uncomment. Or copy the existing canonical file (761f965c). |
| |
| ## Files |
| - `build_oos_test.sh` — driver (build + md5 check) |
| - `_build/{build_oos_v2.py, build_caption_variants.py, ts_align/{data,qa_templates,__init__}.py}` |
| - raw source: `../Benchmarks/TS_Caption_test/{univar,bivar,multivar}/` |
| |
| ## Note |
| QA (`154306d835`) is reproducible only against the current `_build/ts_align/data.py` |
| (599bb34a) prompt pool — pin it if the QA md5 ever drifts. Caption + variants have no |
| prompt-pool dependency. |
| |
