reports/EVAL_REPORT.md

# Kuiper-R1 — Evaluation Report

Held-out benchmark: `Jackrong/DeepSeek-V4-Distill-8000x` (inverted). n=200 rows. Cross-lineage generalization: training teacher **QwQ-32B** (OpenThoughts3) vs benchmark teacher **DeepSeek-V4-Flash** (GLM-5.1 prompts) — disjoint lineages.

Decoding: temperature 0.4, thinking disabled (the reasoning-model base otherwise emits a `<think>` preamble that eats the token budget). Scorer is deterministic (stdlib).

## Results: untouched base vs SFT vs ORPO (release)

| Metric | Untouched base | Kuiper SFT | **Kuiper ORPO (release)** |
|---|---|---|---|
| final-answer preserved (exact) | 0.155 | 0.805 | **0.820** |
| final-answer preserved (soft) | 0.735 | 0.805 | **0.825** |
| format valid (3-tag contract) | 0.980 | 0.850 | **0.880** |
| no invented tool outputs | 1.000 | 1.000 | **1.000** |
| no contradiction | 1.000 | 0.995 | **0.990** |
| no repetition collapse | 0.980 | 0.840 | **0.825** |
| expansion quality (mean) | 0.791 | 0.725 | **0.769** |
| depth-to-concision (mean) | 0.222 | 0.221 | **0.226** |
| semantic consistency w/ bubbles (mean) | 0.708 | 0.515 | **0.535** |
| trace vs reference similarity (mean) | 0.306 | 0.171 | **0.182** |
| overall (mean) | 0.329 | 0.492 | **0.504** |

## Verdict

**The fine-tune's decisive contribution is byte-exact final-answer preservation** (0.155 → 0.820, a 5.3× gain), plus a higher overall score (0.504 vs 0.329). ORPO improves the SFT on format validity, expansion quality, semantic consistency, and answer preservation.

**Honest caveat:** the untouched base still scores higher on raw trace *polish* (format 0.980 vs 0.880; semantic consistency 0.708 vs 0.535). This is expected: Qwythos is a strong instruction-follower and the target format is spelled out in the system prompt, so it already produces well-formed, thoughtful traces — but it does **not** preserve the given final answer (0.155). Kuiper's value is teaching exact preservation while keeping trace quality close. Training on longer (un-truncated) traces and more data would close the remaining quality gap; both are single-GPU-time limited here (see model card).

**Safety metrics** are clean across the board: no invented tool outputs (1.000), no contradiction (~1.0), no repetition collapse (>0.82), and every output is an explicitly labeled `<synthetic_trace>` — no claim of being any proprietary model's real reasoning.

**Release model:** `merged-fp16-orpo` (best overall + best preservation).