Comparing it against Sabi Noema

#1
by vmarcelo - opened

DaisyRose-27B vs Sabi-Noema: Comparison

Both models are reinforcement-learning fine-tunes of Qwen3.8-27B, released under Apache-2.0. They share a base but differ in training data, goals and serving story.

All numbers below are the self-reported by the authors, using different harnesses. No independent evaluation was found in the model cards.

Benchmarks

Each card reports the Qwen3.8-27B base under its own harness, so deltas are only meaningful within a single card.

Benchmark DaisyRose-27B Daisy's base Sabi-Noema Sabi's base
Terminal-Bench 2.1 75.5 73.0 76.0 73.0
DeepSWE 46.2 (v1.1) 42.2 47.5 38.5
Terminal-Bench 4.1 12.0 5.6 n/a n/a
SWE-Bench Pro v2 94.7 91.3 n/a n/a
GPQA Diamond 90.7 89.2 90.6 89.2
IFBench 82.4 79.5 80.9 79.5
HLE (text only) 34.3 33.9 n/a n/a
τ³-bench (/278) n/a n/a 259 220
τ³-bench pass rate n/a n/a 93.2% 79.1%
BFCL v4 multi-turn n/a n/a 63.9 59.6
MCPMark (/55) n/a n/a 48 36
Hallucination rate (lower is better) n/a n/a 15.1% 30.4%
LiveCodeBench v6 n/a n/a 88.0 89.7
MMLU-Pro n/a n/a 85.1 85.4

Reading the numbers

  • Coding benchmarks are roughly tied. 76.0 vs 75.5 on Terminal-Bench 2.1 is within noise, especially across different harnesses.
  • DeepSWE deltas are not comparable. Sabi's +9 points rests on a base score of 38.5 that they measured themselves. Daisy cites 42.2 for the same base, which would shrink Sabi's gain.
  • Sabi retains general capability with small trade-offs: slightly below base on LiveCodeBench v6 and MMLU-Pro.
  • Daisy's strongest claim is on harder, longer tasks (Terminal-Bench 4.1 more than doubles from 5.6 to 12.0).
  • Sabi's strongest claims are in tool calling, agent policy-following and reliability (τ³-bench, MCPMark, hallucination rate).

Efficiency

DaisyRose-27B Sabi-Noema
Output length Long thinking; the card says it can run to tens of thousands of tokens and recommends leaving max_tokens unset Median 29.3K output tokens per Terminal-Bench 2.1 task vs 65.3K for base
Claim None specific to token efficiency 2.3x fewer output tokens than Qwen3.8-27B (and 2.0x fewer than Swift 1.5) on tasks both solve
DeepSWE n/a 82.7K mean output tokens per task vs 118.8K for base

Multi-token prediction (MTP) head

DaisyRose-27B Sabi-Noema
MTP head present Yes: 0.42B parameters, kept unchanged from the base model No: the card says the checkpoint has no MTP head
Practical effect Can use MTP, and RadixArk also ships a separate draft model (RadixArk/Qwen3.8-27B-DSpark) llama.cpp conversion requires convert_hf_to_gguf.py --no-mtp

Because Daisy's MTP head was not retrained during RL, its acceptance rate for speculative decoding may be somewhat lower than on the base model. The card does not report this, which is likely why the authors recommend the dedicated DSpark draft.

I just use a standard MTP Q4 draft head of the base, works well with acceptance rates between 0.5 and 0.95 draft size 4, nothing to worry about.

Also, I think this comparison is not very helpful unfortunately, thanks for sharing anyway. But real-world task comparison is needed, just comparing the reported benchmark values is not enough as the difference is mostly/all noise AFAICT.

Sign up or log in to comment