fengxr93's picture
TS-Align benchmark reproduction bundles + canonical eval data + dataset sources
c3efe57
|
Raw
History Blame Contribute Delete
2.22 kB

OOS test set (TS_Caption_test) — faithful rebuild from raw (used datasets only)

Reproduces the OOS request files we actually use, byte-for-byte, from the raw TS_Caption_test. bash build_oos_test.sh writes exactly these 5:

file what md5
threesplit_501ts_promptV2.jsonl metric-QA (zh) — champion/joint headline 154306d835bb...
caption_501ts_3split.jsonl caption (zh, canonical) 820fe34d2997...
caption_3split_501ts_para1.jsonl caption zh paraphrase #1 11dce0be3dd3...
caption_3split_501ts_para2.jsonl caption zh paraphrase #2 ced4dfcf28b7...
caption_3split_501ts_en.jsonl caption en mirror b4fa39d517e6...

Pipeline

  1. _build/build_oos_v2.py --seed 42 (patched to emit only the 2 used files; the mvonly_* / threesplit_NEWv1 / caption_500ts_mvonly variants are commented out, # [release: unused, not generated]). Reads raw univar/bivar/multivar; QA prompt via ts_align.data.sample_stage2_prompt.
  2. _build/build_caption_variants.py (patched to 3split-only) — swaps the caption prompt with hardcoded para1/para2/en templates (deterministic, no external LLM), all other fields bit-identical.

Deliberately NOT built here

  • mvonly / NEWv1 QA + caption_500ts_mvonly — variants we don't use (not generated).
  • threesplit_..._promptV2_EN.jsonl (QA English) — needs an LLM translation (build_en_qa.py, GLM/Anthropic); can't be byte-rebuilt deterministically. It's an optional (commented) step in the driver: export ANTHROPIC_AUTH_TOKEN=... ANTHROPIC_BASE_URL=... EXTRACTOR_MODEL=glm-5.1 then uncomment. Or copy the existing canonical file (761f965c).

Files

  • build_oos_test.sh — driver (build + md5 check)
  • _build/{build_oos_v2.py, build_caption_variants.py, ts_align/{data,qa_templates,__init__}.py}
  • raw source: ../Benchmarks/TS_Caption_test/{univar,bivar,multivar}/

Note

QA (154306d835) is reproducible only against the current _build/ts_align/data.py (599bb34a) prompt pool — pin it if the QA md5 ever drifts. Caption + variants have no prompt-pool dependency.