File size: 1,612 Bytes
1c0d385 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | # Gold Batch: targeted class-balanced SFT rows (2026-08-13)
Purpose: raise the v22 Spock line from a false-biased "format, not verdict"
state toward the 1,500-3,000 handcrafted-row floor (tiny-model-reasoning).
Each row is the v22 Spock conversational schema:
{"persona":"spock","task":"<class>","user":"...","assistant":
"<|scratchpad|> ... <|final|>I consider this <verdict>. ..."}
Class balance per the verified class-imbalance research (arXiv 2402.19449):
a class's accuracy is what it gets from training, so every verdict class must
have MANY examples. Target: 700 verified rows (true/false/unsubstantiated/
contradiction/overclaim/misleading/abstain ~100 each).
Hard rules (tiny-model-kd / developer's credo):
- Handcrafted, verifiable, teacher-authored. No generators/scripts that
fabricate content — the tooling below only assembles curated JSON from the
typed sources in this directory.
- No class rows authored by machine. Every row is a real claim the model
must reason to a verdict, matching the v22 Spock voice + abstention honesty.
Source files (author these by hand, then assemble):
data/gold_700/true.jsonl, false.jsonl, unsubstantiated.jsonl,
contradiction.jsonl, overclaim.jsonl, misleading.jsonl, abstain.jsonl
Each line = {"user": "...", "assistant": "<|scratchpad|>...<|final|>I consider this ..."}
The fields persona/task are added by data/build_gold_700.py; no class rows
are generated by the script.
Expected outcome gate (tiny-model-eval): after retraining, a real main-battery
score on the new LoRA-SFT v23 must be >= 0.30 before DPO; >= 0.40 is the
release line.
|