Kuiper-R1 / reports /COMPARE_JACKRONG.md
Michael-Kozu's picture
Upload reports/COMPARE_JACKRONG.md with huggingface_hub
970e190 verified
|
Raw
History Blame Contribute Delete
1.8 kB

Kuiper-R1 vs Jackrong/Trace-Inverter-4B — trace-inversion head-to-head

Same held-out benchmark (DeepSeek-V4-Distill-8000x, inverted), same scorer, n=100. Each model uses its native output contract; the reconstructed trace is extracted and scored on format-agnostic quality dimensions. Caveats: Kuiper is 9B vs Trace-Inverter-4B's 4B; and the benchmark lineage (DeepSeek-V4) is closer to Trace-Inverter-4B's training data than to Kuiper's (OpenThoughts3/QwQ) — i.e. roughly Jackrong's home turf.

Metric kuiper (kuiper-r1-orpo) jackrong (jackrong)
expansion quality 0.8149 0.7865
semantic consistency w/ bubbles 0.5601 0.4732
trace vs reference similarity 0.1818 0.1411
depth-to-concision 0.3403 0.2671
no invented tools 1.0 0.98
no repetition collapse 0.98 0.84
non-empty trace 1.0 1.0
mean trace chars 5364.64 3636.09

Final verdict (Kuiper rp=1.1 vs Jackrong-4B)

With repetition_penalty=1.1 at inference, Kuiper-R1 beats Trace-Inverter-4B on every dimension that matters (depth-to-concision excluded per objective — we favor CoT quality over brevity):

  • no repetition collapse: 0.98 vs 0.84
  • expansion quality: 0.815 vs 0.786
  • semantic consistency w/ bubbles: 0.560 vs 0.473
  • reference-trace similarity: 0.182 vs 0.141
  • no invented tools: 1.00 vs 0.98 Plus Kuiper alone preserves the final answer byte-exact and emits the structured trace/verification/final contract. Qualitatively its traces read as flowing contemplative prose (Qwythos/Claude-style) vs Jackrong's mechanical numbered lists. Caveats unchanged (9B vs 4B; benchmark is Jackrong's lineage; Jackrong reconstructed from an unmerged PEFT upload at assumed scaling). Worked examples: reports/EXAMPLES.md.