README.md

---
license: mit
language:
  - en
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-1.7B-Base
base_model_relation: finetune
datasets:
  - alibaba-pai/DistilQwen_100k
metrics:
  - accuracy
  - exact_match
  - code_eval
tags:
  - qwen3
  - qwen
  - qwen3-1.7b
  - qwen3-8b
  - quintus
  - quintus-1.7b
  - causal-lm
  - text-generation
  - language-model
  - chat
  - assistant
  - compact-llm
  - small-language-model
  - knowledge-distillation
  - online-kd
  - full-vocabulary-kd
  - supervised-fine-tuning
  - sft
  - reasoning
  - code-generation
  - english
  - pytorch
  - transformers
  - vllm
widget:
  - text: "Explain knowledge distillation in simple terms."
  - text: "Solve this step by step: If a train travels 180 km in 3 hours, what is its average speed?"
---

# Quintus

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1TdMSN5HzD1mToCFVf_qQoj10NGZLy2V0?usp=sharing)
[![Hugging Face Model](https://img.shields.io/badge/Hugging%20Face-Quintus-ffcc4d?style=flat-square&logo=huggingface&logoColor=yellow)](https://huggingface.co/iamrahulreddy/Quintus)
[![Docs](https://img.shields.io/badge/Docs-Project%20Guide-0f766e?style=flat-square&logo=googledocs&logoColor=white)](docs/index.md)
[![Benchmarks](https://img.shields.io/badge/Benchmarks-Scoreboard-2563eb?style=flat-square&logo=speedtest&logoColor=white)](docs/benchmarks.md)
[![License: MIT](https://img.shields.io/badge/License-MIT-111827?style=flat-square)](LICENSE)
[![Base Model](https://img.shields.io/badge/Base-Qwen3--1.7B--Base-7c3aed?style=flat-square)](https://huggingface.co/Qwen/Qwen3-1.7B-Base)
[![Teacher](https://img.shields.io/badge/Teacher-Qwen3--8B-b45309?style=flat-square)](https://huggingface.co/Qwen/Qwen3-8B)

**Quintus-1.7B** is a compact English-focused assistant built from
`Qwen/Qwen3-1.7B-Base`. The project uses **online full-vocabulary knowledge
distillation** from a `Qwen/Qwen3-8B` teacher, followed by a targeted SFT stage
for assistant behavior, identity grounding, and generation stability.

Final model weights:
[iamrahulreddy/Quintus](https://huggingface.co/iamrahulreddy/Quintus)

## Core Technical Points

- **Dense KD signal:** the final training path streams the teacher's full
  vocabulary distribution live instead of relying on sparse cached top-k logits.
- **Base-student strategy:** the student starts from `Qwen/Qwen3-1.7B-Base`,
  leaving more room for distillation before assistant-format tuning.
- **Assistant-only supervision:** prompt text, chat headers, separators, and
  padding are masked out of the supervised target region.
- **Sequence packing:** deterministic first-fit decreasing packing improves
  useful-token throughput at 4096-token context length.
- **Public benchmark controls:** raw/chat prompt format, metric extraction,
  generation budget, and artifact hygiene are documented explicitly.

## Training Summary

The release training path is a two-stage pipeline:

1. **Online KD:** train the 1.7B base student against live teacher logits from a
   Qwen3-8B teacher.
2. **Targeted SFT:** tune the distilled checkpoint for assistant-style
   interaction, persona consistency, and repetition control.

## Reuse As A KD Framework

Quintus is released as a trained 1.7B assistant, but the repository is also a
reusable reference pipeline for compact-model distillation. The same structure
can be adapted to other teacher/student pairs with changes to the model IDs,
tokenizer, dataset source, local paths, sequence length, batch schedule, and
hardware-specific memory settings in [configs/config.yaml](configs/config.yaml).

The reusable pieces are split across the codebase: assistant-only masking,
sequence packing, online full-vocabulary KD loss, checkpoint/resume metadata,
validation, provenance checks, SFT, and evaluation. The final pattern is:

1. Distill a smaller base student from a stronger teacher with online KD.
2. Apply targeted SFT to recover assistant behavior, formatting, identity, and
   generation stability.

![Quintus Architecture](assets/quintus_architecture.png)

Core KD objective:

$$
\mathcal{L}_{\text{total}}
= \alpha \mathcal{L}_{\text{CE}}
+ (1 - \alpha)\mathcal{L}_{\text{KD}}
$$

For the final run,

$$
\alpha = 0.3,\quad T = 2.0
$$

Configuration snapshot:

| Setting | Value |
| :--- | :--- |
| Teacher | `Qwen/Qwen3-8B` |
| Student | `Qwen/Qwen3-1.7B-Base` |
| Tokenizer | `Qwen/Qwen3-1.7B` |
| Data | ~90K English-only samples from DistilQwen_100k |
| Max sequence length | 4096 |
| Epochs | 1 |
| Learning rate | `5.0e-6` |
| Weight decay | `0.1` |
| Warmup ratio | `0.05` |
| Online KD token chunk | 2048 |
| Micro batch | 4 |
| Gradient accumulation | 2 |
| Sequence packing | enabled, `pack_length = 4096` |
| Attention | FlashAttention-2 when available |
| Liger kernels | enabled for compatible Qwen-family ops |
| Optimizer | fused AdamW |
| `torch.compile` | disabled |
| Gradient checkpointing | disabled |
| Seed | 25 |

> [!NOTE]
> FlashAttention-2, Liger kernels, and fused AdamW are acceleration paths. Keep
> the baseline load path compatible with standard Transformers and vLLM APIs
> before publishing checkpoints. `torch.compile` stayed disabled because this
> KD shape showed high Inductor memory overhead, dynamic-shape graph breaks,
> recompile overhead, and checkpoint portability risk from `_orig_mod.` state
> dict prefixes when compiled modules are not unwrapped before saving.

> [!TIP]
> The B200-oriented defaults are conservative for the 8B teacher to 1.7B
> student workload. Smaller teacher/student pairs may tolerate larger
> micro-batches, but full-vocabulary KD scales sharply with vocabulary width.

The editable run configuration lives in [configs/config.yaml](configs/config.yaml).
Paths and Hub destinations are left as placeholders so each runner can set local
directories and repository names directly.

## Why Online KD Replaced Offline Top-K KD

Earlier experiments cached only the teacher's top-k logits. That made storage
smaller, but with a Qwen vocabulary around 151K tokens, $k = 8$ exposes only:

$$
\frac{k}{|V|}
= \frac{8}{151{,}665}
\approx 5.3 \times 10^{-5}
= 0.0053\%
$$

of the vocabulary support at each position. The sparse signal could perturb the
student, but it did not consistently transfer deeper reasoning behavior.

The final online path keeps the teacher and student in memory together and
computes KL divergence against the teacher's full-vocabulary distribution. Token
chunking keeps that dense objective feasible without materializing a single
large KL workspace.

## Benchmark Scoreboard

The final public scoreboard compares `Qwen/Qwen3-1.7B-Base`,
`Qwen/Qwen3-1.7B-Instruct`, and Quintus-1.7B.

![Model Evaluation Scoreboard](assets/benchmark_scoreboard.png)

The strongest signal is the reasoning crossover: Quintus beats both the base
and official 1.7B instruct model on GSM8K, ARC-Challenge, and WinoGrande while
remaining at the same parameter scale.

See [docs/benchmarks.md](docs/benchmarks.md) for the numeric table and
interpretation. See
[docs/evaluation_methodology.md](docs/evaluation_methodology.md) for benchmark
controls.

## Evaluation Notes

Evaluation uses a mixture of EvalPlus and `lm-evaluation-harness`/vLLM style
benchmarks. The repository keeps evaluation methodology separate because prompt
format can change the result:

- Raw completion comparisons are used for base capability.
- Chat-template comparisons are used for assistant-format behavior.
- Log-likelihood tasks such as ARC-Challenge and PIQA should usually stay raw.
- GSM8K can differ between strict `####` parsing and flexible number
  extraction.
- Metric extraction must ignore `stderr`, aliases, and wrong filter keys.
- Runtime versions, checkpoint identity, generation budget, and stale output
  cleanup are part of the evaluation contract.

The active benchmark runner is [sft/evaluate.py](sft/evaluate.py). It covers
EvalPlus code tasks and `lm-evaluation-harness`/vLLM tasks, including GSM8K
10-shot evaluation with an extended generation budget.

## Repository Map

```text
configs/        Public run profile and DeepSpeed Zero-2 template.
src/            Data prep, online KD, losses, packing, checkpoints, provenance.
sft/            Post-KD SFT, local chat, and consolidated evaluation runner.
docs/           Public architecture, training, evaluation, and release notes.
weight_audit/   Checkpoint structure and weight-divergence audit material.
```

Key files:

- [src/train.py](src/train.py): SFT, offline KD compatibility, and final
  `online_kd` training entry point.
- [src/download.py](src/download.py): model setup, dataset loading, schema
  normalization, tokenization, and assistant-only loss masks.
- [src/losses.py](src/losses.py): CE/KD objective, including online full-vocab
  KD token chunking.
- [src/sequence_packing.py](src/sequence_packing.py): deterministic first-fit
  decreasing sequence packing.
- [src/checkpoints.py](src/checkpoints.py): checkpoint save/resume metadata and
  packing compatibility checks.
- [src/provenance.py](src/provenance.py): tokenizer/model/data contract checks.
- [sft/train_sft.py](sft/train_sft.py): post-KD supervised fine-tuning.
- [sft/evaluate.py](sft/evaluate.py): EvalPlus and
  `lm-evaluation-harness`/vLLM benchmark runner.
- [sft/chat.py](sft/chat.py): local interactive chat wrapper.

## Commands

Install the base dependencies:

```bash
pip install -r requirements.txt
```

For training and benchmark runs, install the matching extras:

```bash
pip install -r requirements-train.txt
pip install -r requirements-eval.txt
```

Inspect or prepare data/model assets:

```bash
python -m src.download --help
```

Run the final KD path after editing [configs/config.yaml](configs/config.yaml)
for local paths and hardware:

```bash
python -m src.train --phase online_kd
```

Hub checkpoint uploads are off by default for local runs. Pass
`--upload_last_checkpoint` or the step/epoch upload flags only after setting the
target repository and `HF_TOKEN`.

Run the consolidated benchmark suite:

```bash
python sft/evaluate.py
```

Start local chat with a downloaded or local checkpoint:

```bash
python sft/chat.py --model_path path/to/quintus/checkpoint
```

## Interactive Chat

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

PUBLIC_REPO_ID = "iamrahulreddy/Quintus"

print(f"Loading Quintus from {PUBLIC_REPO_ID}...")
tokenizer = AutoTokenizer.from_pretrained(PUBLIC_REPO_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    PUBLIC_REPO_ID,
    device_map="auto",
    dtype=torch.float16,
    trust_remote_code=True,
)

stop_tokens = ["<|endoftext|>", "<|im_end|>"]
eos_token_ids = [tokenizer.eos_token_id] if tokenizer.eos_token_id is not None else []
for token in stop_tokens:
    token_id = tokenizer.convert_tokens_to_ids(token)
    if token_id is not None and token_id not in eos_token_ids:
        eos_token_ids.append(token_id)

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

conversation_history = [
    {
        "role": "system",
        "content": (
            "You are Quintus, a highly capable AI assistant created by "
            "Muskula Rahul. You are helpful, precise, and logically sound."
        ),
    }
]

print()
print("Quintus Chat (type 'quit' to exit)")
print()

while True:
    try:
        user_input = input("You: ").strip()
        if user_input.lower() in ["quit", "exit"]:
            print("\nGoodbye!")
            break
        if not user_input:
            continue

        conversation_history.append({"role": "user", "content": user_input})

        prompt = tokenizer.apply_chat_template(
            conversation_history,
            tokenize=False,
            add_generation_prompt=True,
        )

        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

        print("Quintus: ", end="", flush=True)

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=512,
                temperature=0.7,
                top_p=0.9,
                do_sample=True,
                streamer=streamer,
                pad_token_id=tokenizer.eos_token_id,
                eos_token_id=eos_token_ids,
            )

        generated_ids = outputs[0][inputs.input_ids.shape[-1]:]
        assistant_response = tokenizer.decode(
            generated_ids,
            skip_special_tokens=True,
        ).strip()
        conversation_history.append({"role": "assistant", "content": assistant_response})
        print()

    except KeyboardInterrupt:
        print("\n\nGoodbye!")
        break
```

## Documentation

- [Documentation Index](docs/index.md): recommended public reading order.
- [Architecture](docs/architecture.md): end-to-end data flow, modules, and
  training phases.
- [Experiment Timeline](docs/experiment_timeline.md): why the project moved
  from offline top-k KD to online full-vocabulary KD.
- [Training Playbook](docs/training_playbook.md): memory rules, packing,
  kernels, checkpointing, and B200-oriented guidance.
- [Pipeline Hardening](docs/pipeline_hardening.md): silent-failure classes,
  artifact contracts, and safety checks.
- [Evaluation Methodology](docs/evaluation_methodology.md): raw/chat controls,
  parser traps, metric extraction, and qualitative evaluation rules.
- [Engineering Insights](docs/engineering_insights.md): condensed lessons and
  design decisions.
- [Benchmarks](docs/benchmarks.md): verified scoreboard and interpretation.
- [Weight Audit](docs/weight_audit.md): structural checkpoint sanity checks and
  weight-divergence summary.
- [Hugging Face Model Card](docs/huggingface_model_card.md): release-page
  copy for the public model card.

## Limitations

- Quintus is still a 1.7B model and inherits compact-model capacity limits.
- Factual answers can be confidently wrong and should be verified.
- Code generation may still contradict stated complexity or edge-case
  requirements.
- Raw and chat-template results are not interchangeable.
- Additional preference tuning or DPO would likely improve calibration, refusal
  behavior, and open-ended assistant polish.

## Credits

Quintus builds on open model, dataset, and tooling work from the broader LLM
community:

- [Qwen Team](https://qwenlm.github.io/) and the
  [Qwen Hugging Face organization](https://huggingface.co/Qwen) for the Qwen3
  model family.
- [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B), used as the
  distillation teacher.
- [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base), used
  as the base student checkpoint.
- [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B), used for the
  tokenizer and chat-template contract.
- [Alibaba PAI](https://huggingface.co/alibaba-pai) for the
  [`DistilQwen_100k`](https://huggingface.co/datasets/alibaba-pai/DistilQwen_100k)
  dataset used as the primary instruction source after filtering.
- [Hugging Face Transformers](https://github.com/huggingface/transformers) for
  model loading, tokenization, and generation APIs.
- [vLLM](https://github.com/vllm-project/vllm),
  [EvalPlus](https://github.com/evalplus/evalplus), and
  [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
  for evaluation infrastructure.
- [FlashAttention](https://github.com/Dao-AILab/flash-attention) and
  [Liger Kernel](https://github.com/linkedin/Liger-Kernel) for performance
  kernels used or validated during training.

## License And Author

This software is distributed under the MIT License. Refer to the
[LICENSE](LICENSE) file for full text.

Author: Muskula Rahul - [@iamrahulreddy](https://github.com/iamrahulreddy)

## Citation

If this model, codebase, or training pipeline is useful in your work, please cite this repository and acknowledge the upstream Qwen3 models.