docs/benchmarks.md

# Benchmarks

The release scoreboard compares Qwen3-1.7B-Base, Qwen3-1.7B-Instruct, and Quintus-1.7B. Evaluations use a mixture of EvalPlus and lm-evaluation-harness style benchmarks, with greedy or deterministic settings where applicable.

For the detailed benchmark-control rules, see [Evaluation Methodology](evaluation_methodology.md).

## Final Scoreboard

| Benchmark | Qwen3-1.7B-Base | Qwen3-1.7B-Instruct | Quintus-1.7B |
| :--- | :---: | :---: | :---: |
| HumanEval pass@1 | 67.1% | 70.7% | 67.7% |
| MBPP pass@1 | 67.2% | 58.2% | 64.8% |
| GSM8K, 10-shot flexible | 69.98% | 69.75% | 74.30% |
| ARC-Challenge acc_norm | 55.72% | 52.99% | 58.36% |
| WinoGrande, 5-shot | 65.67% | 61.01% | 66.38% |
| PIQA acc_norm | 75.63% | 72.09% | 75.57% |

## Full Checkpoint Matrix

The compact scoreboard above is the headline comparison. The full matrix below
records the broader evaluation suite across four checkpoints:

- `Base`: `Qwen/Qwen3-1.7B-Base`
- `Instruct`: `Qwen/Qwen3-1.7B-Instruct`
- `Pre-SFT`: online KD checkpoint before targeted SFT
- `Quintus SFT`: final public Quintus checkpoint

$\Delta$ vs Instruct is computed as Quintus SFT minus
`Qwen/Qwen3-1.7B-Instruct`, in percentage points. GSM8K strict and flexible
scores are listed separately because parser behavior and EOS handling can
change the measured result.

| Area | Benchmark | Base | Instruct | Pre-SFT | Quintus SFT | $\Delta$ vs Instruct |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| Coding | HumanEval pass@1 | 67.1% | 70.7% | 68.3% | 67.7% | -3.0 pp |
| Coding | HumanEval+ | 60.4% | 64.0% | 62.8% | 60.4% | -3.6 pp |
| Coding | MBPP pass@1 | 67.2% | 58.2% | 63.0% | 64.8% | +6.6 pp |
| Coding | MBPP+ | 58.2% | 50.0% | 54.5% | 56.3% | +6.3 pp |
| Math | GSM8K flexible | 70.0% | 69.8% | 74.4% | 74.3% | +4.5 pp |
| Math | GSM8K strict | 69.6% | 69.8% | 74.1% | 60.9% | -8.9 pp |
| Reasoning/commonsense | WinoGrande, 5-shot | 65.7% | 61.0% | 66.0% | 66.4% | +5.4 pp |
| Reasoning/commonsense | ARC-Challenge acc | 51.5% | 49.5% | 51.9% | 54.8% | +5.3 pp |
| Reasoning/commonsense | ARC-Challenge acc_norm | 55.7% | 53.0% | 55.6% | 58.4% | +5.4 pp |
| Reasoning/commonsense | BoolQ | 79.0% | 77.5% | 77.3% | 71.6% | -5.9 pp |
| Reasoning/commonsense | PIQA acc | 75.6% | 72.9% | 75.8% | 75.2% | +2.3 pp |
| Reasoning/commonsense | PIQA acc_norm | 75.6% | 72.1% | 75.7% | 75.6% | +3.5 pp |

## Interpretation

The strongest result is the reasoning crossover: Quintus beats both the base and the official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande, despite remaining at the same parameter scale.

The coding picture is mixed but useful:

- HumanEval remains slightly below Qwen3-1.7B-Instruct.
- MBPP is substantially above Qwen3-1.7B-Instruct, though still below the base model.

This suggests the model gained useful instruction-following and reasoning behavior without fully matching larger or more heavily aligned code-specialized models.

## What The Benchmarks Support

These results support four claims:

1. Online KD transferred reasoning capability into a compact student.
2. The final model did not merely memorize assistant formatting; it improved several reasoning and commonsense metrics.
3. SFT helped expose the distilled capability in an assistant setting.
4. The model still has capacity limits typical of the 1.7B scale, especially on code execution reliability and long multi-step algorithm generation.

## Evaluation Caveats

Benchmark comparisons are sensitive to prompt format. Raw completion, chat-template generation, and log-likelihood multiple-choice scoring can produce different rankings. For fair interpretation:

- Compare raw models against raw models when measuring base reasoning.
- Compare chat-wrapped models against chat-wrapped models when measuring format alignment.
- Treat open-ended qualitative prompts as alignment tests, not as a replacement for standardized benchmarks.

Important implementation caveats:

- GSM8K extraction can differ between strict `####` parsing and flexible number extraction.
- Multiple-choice log-likelihood tasks can be distorted by chat templates.
- `acc_norm` is preferred when answer-option length bias can change the ranking.
- Metric extraction scripts must reject `stderr` and `alias` fields when looking for the actual score.
- Runtime versions should be recorded with benchmark outputs because harness behavior can change across releases.

## Earlier Development Signals

Before the final Qwen3 8B -> 1.7B run, earlier experiments showed that sparse offline top-k KD could not consistently outperform strong baselines. Those runs were useful because they identified the bottleneck: sparse cached teacher logits were not dense enough to transfer deeper reasoning pathways.

The final move to online full-vocabulary KD is the key methodological change behind the stronger final results.