File size: 2,217 Bytes
c3efe57 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | # OOS test set (TS_Caption_test) — faithful rebuild from raw (used datasets only)
Reproduces the OOS request files **we actually use**, byte-for-byte, from the raw
`TS_Caption_test`. `bash build_oos_test.sh` writes exactly these 5:
| file | what | md5 |
|---|---|---|
| `threesplit_501ts_promptV2.jsonl` | metric-QA (zh) — champion/joint headline | `154306d835bb...` |
| `caption_501ts_3split.jsonl` | caption (zh, canonical) | `820fe34d2997...` |
| `caption_3split_501ts_para1.jsonl`| caption zh paraphrase #1 | `11dce0be3dd3...` |
| `caption_3split_501ts_para2.jsonl`| caption zh paraphrase #2 | `ced4dfcf28b7...` |
| `caption_3split_501ts_en.jsonl` | caption en mirror | `b4fa39d517e6...` |
## Pipeline
1. **`_build/build_oos_v2.py` --seed 42** (patched to emit **only** the 2 used files;
the mvonly_* / threesplit_NEWv1 / caption_500ts_mvonly variants are commented out,
`# [release: unused, not generated]`). Reads raw univar/bivar/multivar; QA prompt
via `ts_align.data.sample_stage2_prompt`.
2. **`_build/build_caption_variants.py`** (patched to 3split-only) — swaps the caption
prompt with hardcoded para1/para2/en templates (deterministic, **no external LLM**),
all other fields bit-identical.
## Deliberately NOT built here
- **mvonly / NEWv1 QA + caption_500ts_mvonly** — variants we don't use (not generated).
- **`threesplit_..._promptV2_EN.jsonl` (QA English)** — needs an LLM translation
(`build_en_qa.py`, GLM/Anthropic); can't be byte-rebuilt deterministically. It's an
optional (commented) step in the driver: `export ANTHROPIC_AUTH_TOKEN=... ANTHROPIC_BASE_URL=...
EXTRACTOR_MODEL=glm-5.1` then uncomment. Or copy the existing canonical file (761f965c).
## Files
- `build_oos_test.sh` — driver (build + md5 check)
- `_build/{build_oos_v2.py, build_caption_variants.py, ts_align/{data,qa_templates,__init__}.py}`
- raw source: `../Benchmarks/TS_Caption_test/{univar,bivar,multivar}/`
## Note
QA (`154306d835`) is reproducible only against the current `_build/ts_align/data.py`
(599bb34a) prompt pool — pin it if the QA md5 ever drifts. Caption + variants have no
prompt-pool dependency.
|