Kuiper-R1 / reports /EVAL_REPORT.md
Michael-Kozu's picture
Upload reports/EVAL_REPORT.md with huggingface_hub
f8fb1ff verified
|
Raw
History Blame Contribute Delete
2.47 kB

Kuiper-R1 — Evaluation Report

Held-out benchmark: Jackrong/DeepSeek-V4-Distill-8000x (inverted). n=200 rows. Cross-lineage generalization: training teacher QwQ-32B (OpenThoughts3) vs benchmark teacher DeepSeek-V4-Flash (GLM-5.1 prompts) — disjoint lineages.

Decoding: temperature 0.4, thinking disabled (the reasoning-model base otherwise emits a <think> preamble that eats the token budget). Scorer is deterministic (stdlib).

Results: untouched base vs SFT vs ORPO (release)

Metric Untouched base Kuiper SFT Kuiper ORPO (release)
final-answer preserved (exact) 0.155 0.805 0.820
final-answer preserved (soft) 0.735 0.805 0.825
format valid (3-tag contract) 0.980 0.850 0.880
no invented tool outputs 1.000 1.000 1.000
no contradiction 1.000 0.995 0.990
no repetition collapse 0.980 0.840 0.825
expansion quality (mean) 0.791 0.725 0.769
depth-to-concision (mean) 0.222 0.221 0.226
semantic consistency w/ bubbles (mean) 0.708 0.515 0.535
trace vs reference similarity (mean) 0.306 0.171 0.182
overall (mean) 0.329 0.492 0.504

Verdict

The fine-tune's decisive contribution is byte-exact final-answer preservation (0.155 → 0.820, a 5.3× gain), plus a higher overall score (0.504 vs 0.329). ORPO improves the SFT on format validity, expansion quality, semantic consistency, and answer preservation.

Honest caveat: the untouched base still scores higher on raw trace polish (format 0.980 vs 0.880; semantic consistency 0.708 vs 0.535). This is expected: Qwythos is a strong instruction-follower and the target format is spelled out in the system prompt, so it already produces well-formed, thoughtful traces — but it does not preserve the given final answer (0.155). Kuiper's value is teaching exact preservation while keeping trace quality close. Training on longer (un-truncated) traces and more data would close the remaining quality gap; both are single-GPU-time limited here (see model card).

Safety metrics are clean across the board: no invented tool outputs (1.000), no contradiction (~1.0), no repetition collapse (>0.82), and every output is an explicitly labeled <synthetic_trace> — no claim of being any proprietary model's real reasoning.

Release model: merged-fp16-orpo (best overall + best preservation).