Experiment Timeline
This timeline explains why the final Quintus design looks the way it does. It focuses on the technical evolution from sparse offline distillation to the final online full-vocabulary pipeline.
1. Offline Top-K KD Prototype
The earliest design precomputed teacher logits to disk and trained the student from cached top-k supports.
Why it was attractive:
- Avoided loading teacher and student together.
- Reduced KD memory from full vocabulary to top-k support.
- Made cloud interruptions easier to survive because teacher logits were already saved.
Main lessons:
- Serialization contracts matter as much as loss math.
- Top-k token IDs need safe dtypes.
- Teacher-logit shards must preserve original row order.
- Missing or stale shards should fail loudly.
2. Static Audit And Fail-Fast Hardening
The project then moved through a static-audit phase focused on silent failure modes.
Major hardening themes:
- Dataset zero-retention checks.
- Missing-shard hard failures.
- Stale artifact cleanup.
- DeepSpeed accumulation correctness.
- Rank-safe writes.
- Explicit model revision and remote-code policy.
- Stronger provenance metadata.
This phase turned the code from a script bundle into a more reliable training pipeline.
3. Assistant-Only Supervision
The tokenization path originally risked supervising the whole conversation. That can over-train prompts, headers, and formatting tokens.
The corrected path derives loss_mask and trains only on assistant response tokens.
This changed the training contract:
- Prompt tokens provide context.
- Assistant tokens receive CE and KD loss.
- Rows without assistant targets are rejected.
- Checkpoints and datasets must agree on the mask schema.
4. Top-K Plus Residual Bucket
A later offline-KD pass improved the sparse support by adding an "other" bucket for teacher probability mass outside top-k.
This fixed a mathematical weakness: the student should be normalized against the full vocabulary before comparison, not only inside top-k. The residual bucket made offline KD less wrong, but it still compressed most of the teacher distribution into one scalar.
That design was useful, but not enough for flagship results.
5. Dataset And Objective Mismatch
Smoke runs showed a pattern that became important later: held-out KD validation loss can improve while benchmark quality worsens.
Key diagnosis:
- Matching teacher token distributions on a training corpus is not identical to improving GSM8K, ARC, coding, or open-ended assistant quality.
- Dataset order and first-N streaming can bias sample selection.
- Long reasoning traces can overweight style and process tokens relative to final answers.
- Small students can forget useful baseline behavior when full-parameter training is too aggressive.
This motivated stricter downstream evaluation gates.
6. Base Student Pivot
Several runs tested whether distilling into an already-instruct-tuned student caused destructive interference. The base-student hypothesis was sound: a raw base model has more plasticity and fewer alignment paths to overwrite.
The result was only a marginal improvement under offline top-k KD. That was the decisive clue.
Conclusion:
The student choice was not the main bottleneck. Offline top-k sparsity was the main bottleneck.
7. Offline Top-K Ceiling
With $k = 8$, the student saw only a tiny fraction of the teacher vocabulary distribution per target token:
Different $\alpha$ values, epochs, and student initializations did not remove this limit.
Offline top-k KD could perturb the student and sometimes improve narrow metrics, but it could not reliably transfer the teacher's broader reasoning distribution.
The project stopped treating offline top-k KD as the path to a flagship model.
8. Online Full-Vocabulary KD
Online KD became the final architecture.
Instead of reading cached teacher shards, the training loop loads a frozen teacher and runs live teacher forward passes beside the student. The KD loss uses the teacher's full-vocabulary distribution.
Benefits:
- No top-k sparsity ceiling.
- No shard-order mismatch risk.
- No stale teacher-logit cache.
- Stronger transfer signal for reasoning.
Cost:
- Higher VRAM footprint.
- Teacher and student must fit together.
- KL computation needs chunking.
- Throughput depends heavily on packing and kernels.
9. Sequence Packing And B200 Tuning
Sequence packing converted padding waste into useful tokens.
The packing implementation:
- Packs only training data.
- Keeps validation easier to interpret.
- Uses fixed 4096-token bins.
- Inserts masked EOS separators.
- Stores packing metadata in checkpoints.
- Rejects packed/unpacked resume mismatches.
Development probes showed the expected utilization improvement and made online KD fast enough for serious single-GPU runs.
10. English-Only Final Data
The release run focuses on English samples.
Reasons:
- Reduce language drift in open-ended outputs.
- Keep the model's assistant behavior aligned with the intended release language.
- Make qualitative evaluation cleaner.
- Avoid CJK continuation artifacts after missed EOS.
The tradeoff is real: removing multilingual data can reduce access to some reasoning traces. For a public English assistant, language stability is worth that tradeoff.
11. Targeted SFT After KD
Online KD transferred capability, but raw KD is not a full assistant-alignment process.
Targeted SFT was added after KD to improve:
- identity grounding,
- chat format stability,
- practical assistant style,
- repetition control,
- response presentation.
This created the final two-stage public model:
Qwen3-1.7B-Base
-> online full-vocab KD from Qwen3-8B
-> targeted SFT
-> Quintus-1.7B
12. Release Verification
The final release surface combines:
- benchmark scoreboard,
- architecture documentation,
- evaluation methodology notes,
- pipeline hardening notes,
- weight audit,
- model-card draft.
The public docs focus on reusable methods, release results, and reproducible checks.