---
license: apache-2.0
language:
- en
tags:
- distillation
- coding
- agentic
- qwen2.5
- kimi-k3
- gpt-5.6
- fable-5
- frontier-models
- qlora
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
pipeline_tag: text-generation
library_name: transformers
datasets:
- Siddh07ETH/Atlas-Frontier-Model-Traces
---

# 🧬 Atlas-Frontier-Distill-3B

> **An experimental 3B coding model distilled from frontier model traces (Kimi-K3, GPT-5.6, Fable-5).**

<p align="center">
  <img src="./banner.png" width="90%">
</p>

---

# Model Description

**Atlas-Frontier-Distill-3B** is a specialized coding assistant built on top of **Qwen2.5-Coder-3B-Instruct**. The project explores whether the coding and debugging capabilities of large frontier models can be transferred into an efficient 3B parameter model suitable for local and edge deployment.

The model was trained using **QLoRA** on **15,746** carefully curated coding conversations extracted from multiple frontier teacher models. Rather than imitating every response, the dataset was aggressively filtered to preserve only successful reasoning traces and high-quality coding solutions.

---

# Distillation Experiment

This model is part of an empirical research project by **Pluto AI Research Lab** investigating knowledge distillation through curated execution traces.

During dataset construction, raw frontier model outputs contained a significant amount of unusable samples including:

- Empty tool calls
- Failed API responses
- Placeholder outputs
- "Needs human" responses
- Abandoned reasoning chains

Over **20,375** low-quality conversations were removed, leaving **15,746** high-quality coding traces for training.

The objective was to ensure the student model learns productive coding behavior instead of failure patterns.

---

> [!NOTE]
> ## 🔬 Behavioral Delta *(via llm-diff)*
> Behavioral regression analysis comparing the **base model** against **Atlas-Frontier-Distill-3B** using **Pluto AI's open-source `llm-diff`** behavioral evaluation tool.

```bash
llm-diff ollama/qwen2.5-coder:3b ollama/atlas-frontier-distill-3b --backend ollama
```

| Metric | Result | Observation |
|:-------|:------:|:------------|
| 🎯 **Instruction Fidelity** | **1.00** | Maintained at 1.00. The distillation traces did not break the model's ability to follow strict formatting constraints. |
| ⚡ **Response Style** | **Improved** | Total word count remained stable, but GPT-4 style filler preamble words were reduced to 0. The frontier traces taught the model to output pure code immediately without conversational bloat |
| 🧠 **Reasoning Consistency** | **1.00** | Maintained at 1.00. The model successfully retained its logical consistency across reframed syllogisms. |

> **Key Takeaway**

> Want to audit your own model upgrades? Install **llm-diff** today: **pip install pluto-llm-diff**


# Dataset Processing

The training corpus underwent a dedicated preprocessing pipeline:

- ✅ Schema unification from multiple chat formats into ChatML
- ✅ Automatic extraction of user/assistant conversations
- ✅ Removal of invalid tool outputs
- ✅ Deduplication
- ✅ Conversation validation
- ✅ Coding-only filtering
- ✅ High-quality reasoning preservation

Final training corpus:

- **15,746 conversations**
- **~36 MB cleaned dataset**
- **100% coding & debugging focused**

Dataset:

> **Siddh07ETH/Atlas-Frontier-Model-Traces**

---

# Training Details

| Property | Value |
|-----------|-------|
| Base Model | Qwen/Qwen2.5-Coder-3B-Instruct |
| Parameters | 3.09B |
| Training Method | QLoRA (NF4 4-bit) |
| LoRA Rank | r=32 |
| LoRA Alpha | 64 |
| LoRA Dropout | 0.05 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Optimizer | Paged AdamW 8-bit |
| Learning Rate | 1e-4 |
| Scheduler | Cosine |
| Sequence Length | 1024 |
| Epochs | 1 |
| Training Steps | 493 |
| Final Training Loss | **1.7056** |
| Hardware | Kaggle Tesla T4 (16 GB) |
| Framework | Transformers + PEFT + TRL |

---

# Evaluation

<p align="center">
  <img src="./benchmark.png" width="90%">
</p>

### 🧪 Benchmark Results (HumanEval Pass@1)
Evaluated on a random subset of 20 complex HumanEval problems using greedy decoding (`temperature=0.0`) to test pure reasoning capabilities.

| Model | Pass@1 Accuracy |
| :--- | :---: |
| Base Model (Qwen2.5-Coder-3B-Instruct) | 60.0% |
| **Atlas-Frontier-Distill-3B** | **65.0%** |

> **Key Takeaway:** The fine-tuning process successfully improved the model's ability to solve complex edge-case logic problems (such as Problem #10 in our evaluation subset) while maintaining **zero regression** on tasks the base model already solved correctly. This validates the distillation of high-quality frontier reasoning traces into the 3B parameter space.

---

# Intended Use

Atlas-Frontier-Distill-3B is intended for:

- Local coding assistants
- IDE integration
- Autonomous debugging agents
- Python code generation
- GGUF deployment
- Ollama deployment
- Edge inference
- Software engineering research

---

# Limitations

- Optimized specifically for coding and debugging tasks.
- May perform worse than the base model on open-domain conversation.
- Not trained for creative writing or general-purpose chat.
- Distillation quality depends entirely on the quality of teacher traces.

---

# Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Siddh07ETH/Atlas-Frontier-Distill-3B",
    torch_dtype=torch.float16,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(
    "Siddh07ETH/Atlas-Frontier-Distill-3B"
)

messages = [
    {
        "role": "user",
        "content": "Write a Python function to connect to a PostgreSQL database."
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.2,
        do_sample=True,
    )

print(
    tokenizer.decode(
        outputs[0][inputs.input_ids.shape[1]:],
        skip_special_tokens=True,
    )
)
```

---

# Citation

If you use this model or the accompanying dataset in your research, please cite:

```bibtex
@misc{atlasfrontierdistill3b,
  author    = {Siddharth N.R.},
  title     = {Atlas-Frontier-Distill-3B: Distilling Frontier Model Traces into Edge-Deployable LLMs},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/Siddh07ETH/Atlas-Frontier-Distill-3B}
}
```

---

# Author

**Siddharth N.R.**

**Pluto AI Research Lab**

---

## License

Apache-2.0