| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - distillation |
| - coding |
| - agentic |
| - qwen2.5 |
| - kimi-k3 |
| - gpt-5.6 |
| - fable-5 |
| - frontier-models |
| - qlora |
| base_model: Qwen/Qwen2.5-Coder-3B-Instruct |
| pipeline_tag: text-generation |
| library_name: transformers |
| datasets: |
| - Siddh07ETH/Atlas-Frontier-Model-Traces |
| --- |
| |
| # 🧬 Atlas-Frontier-Distill-3B |
|
|
| > **An experimental 3B coding model distilled from frontier model traces (Kimi-K3, GPT-5.6, Fable-5).** |
|
|
| <p align="center"> |
| <img src="./banner.png" width="90%"> |
| </p> |
|
|
| --- |
|
|
| # Model Description |
|
|
| **Atlas-Frontier-Distill-3B** is a specialized coding assistant built on top of **Qwen2.5-Coder-3B-Instruct**. The project explores whether the coding and debugging capabilities of large frontier models can be transferred into an efficient 3B parameter model suitable for local and edge deployment. |
|
|
| The model was trained using **QLoRA** on **15,746** carefully curated coding conversations extracted from multiple frontier teacher models. Rather than imitating every response, the dataset was aggressively filtered to preserve only successful reasoning traces and high-quality coding solutions. |
|
|
| --- |
|
|
| # Distillation Experiment |
|
|
| This model is part of an empirical research project by **Pluto AI Research Lab** investigating knowledge distillation through curated execution traces. |
|
|
| During dataset construction, raw frontier model outputs contained a significant amount of unusable samples including: |
|
|
| - Empty tool calls |
| - Failed API responses |
| - Placeholder outputs |
| - "Needs human" responses |
| - Abandoned reasoning chains |
|
|
| Over **20,375** low-quality conversations were removed, leaving **15,746** high-quality coding traces for training. |
|
|
| The objective was to ensure the student model learns productive coding behavior instead of failure patterns. |
|
|
| --- |
|
|
| > [!NOTE] |
| > ## 🔬 Behavioral Delta *(via llm-diff)* |
| > Behavioral regression analysis comparing the **base model** against **Atlas-Frontier-Distill-3B** using **Pluto AI's open-source `llm-diff`** behavioral evaluation tool. |
|
|
| ```bash |
| llm-diff ollama/qwen2.5-coder:3b ollama/atlas-frontier-distill-3b --backend ollama |
| ``` |
|
|
| | Metric | Result | Observation | |
| |:-------|:------:|:------------| |
| | 🎯 **Instruction Fidelity** | **1.00** | Maintained at 1.00. The distillation traces did not break the model's ability to follow strict formatting constraints. | |
| | ⚡ **Response Style** | **Improved** | Total word count remained stable, but GPT-4 style filler preamble words were reduced to 0. The frontier traces taught the model to output pure code immediately without conversational bloat | |
| | 🧠 **Reasoning Consistency** | **1.00** | Maintained at 1.00. The model successfully retained its logical consistency across reframed syllogisms. | |
|
|
| > **Key Takeaway** |
|
|
| > Want to audit your own model upgrades? Install **llm-diff** today: **pip install pluto-llm-diff** |
|
|
|
|
| # Dataset Processing |
|
|
| The training corpus underwent a dedicated preprocessing pipeline: |
|
|
| - ✅ Schema unification from multiple chat formats into ChatML |
| - ✅ Automatic extraction of user/assistant conversations |
| - ✅ Removal of invalid tool outputs |
| - ✅ Deduplication |
| - ✅ Conversation validation |
| - ✅ Coding-only filtering |
| - ✅ High-quality reasoning preservation |
|
|
| Final training corpus: |
|
|
| - **15,746 conversations** |
| - **~36 MB cleaned dataset** |
| - **100% coding & debugging focused** |
|
|
| Dataset: |
|
|
| > **Siddh07ETH/Atlas-Frontier-Model-Traces** |
|
|
| --- |
|
|
| # Training Details |
|
|
| | Property | Value | |
| |-----------|-------| |
| | Base Model | Qwen/Qwen2.5-Coder-3B-Instruct | |
| | Parameters | 3.09B | |
| | Training Method | QLoRA (NF4 4-bit) | |
| | LoRA Rank | r=32 | |
| | LoRA Alpha | 64 | |
| | LoRA Dropout | 0.05 | |
| | Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | |
| | Optimizer | Paged AdamW 8-bit | |
| | Learning Rate | 1e-4 | |
| | Scheduler | Cosine | |
| | Sequence Length | 1024 | |
| | Epochs | 1 | |
| | Training Steps | 493 | |
| | Final Training Loss | **1.7056** | |
| | Hardware | Kaggle Tesla T4 (16 GB) | |
| | Framework | Transformers + PEFT + TRL | |
| |
| --- |
| |
| # Evaluation |
| |
| <p align="center"> |
| <img src="./benchmark.png" width="90%"> |
| </p> |
| |
| ### 🧪 Benchmark Results (HumanEval Pass@1) |
| Evaluated on a random subset of 20 complex HumanEval problems using greedy decoding (`temperature=0.0`) to test pure reasoning capabilities. |
| |
| | Model | Pass@1 Accuracy | |
| | :--- | :---: | |
| | Base Model (Qwen2.5-Coder-3B-Instruct) | 60.0% | |
| | **Atlas-Frontier-Distill-3B** | **65.0%** | |
| |
| > **Key Takeaway:** The fine-tuning process successfully improved the model's ability to solve complex edge-case logic problems (such as Problem #10 in our evaluation subset) while maintaining **zero regression** on tasks the base model already solved correctly. This validates the distillation of high-quality frontier reasoning traces into the 3B parameter space. |
| |
| --- |
| |
| # Intended Use |
| |
| Atlas-Frontier-Distill-3B is intended for: |
| |
| - Local coding assistants |
| - IDE integration |
| - Autonomous debugging agents |
| - Python code generation |
| - GGUF deployment |
| - Ollama deployment |
| - Edge inference |
| - Software engineering research |
| |
| --- |
| |
| # Limitations |
| |
| - Optimized specifically for coding and debugging tasks. |
| - May perform worse than the base model on open-domain conversation. |
| - Not trained for creative writing or general-purpose chat. |
| - Distillation quality depends entirely on the quality of teacher traces. |
| |
| --- |
| |
| # Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "Siddh07ETH/Atlas-Frontier-Distill-3B", |
| torch_dtype=torch.float16, |
| device_map="auto", |
| ) |
| |
| tokenizer = AutoTokenizer.from_pretrained( |
| "Siddh07ETH/Atlas-Frontier-Distill-3B" |
| ) |
| |
| messages = [ |
| { |
| "role": "user", |
| "content": "Write a Python function to connect to a PostgreSQL database." |
| } |
| ] |
| |
| text = tokenizer.apply_chat_template( |
| messages, |
| tokenize=False, |
| add_generation_prompt=True, |
| ) |
| |
| inputs = tokenizer(text, return_tensors="pt").to(model.device) |
|
|
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=256, |
| temperature=0.2, |
| do_sample=True, |
| ) |
| |
| print( |
| tokenizer.decode( |
| outputs[0][inputs.input_ids.shape[1]:], |
| skip_special_tokens=True, |
| ) |
| ) |
| ``` |
| |
| --- |
| |
| # Citation |
| |
| If you use this model or the accompanying dataset in your research, please cite: |
| |
| ```bibtex |
| @misc{atlasfrontierdistill3b, |
| author = {Siddharth N.R.}, |
| title = {Atlas-Frontier-Distill-3B: Distilling Frontier Model Traces into Edge-Deployable LLMs}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| url = {https://huggingface.co/Siddh07ETH/Atlas-Frontier-Distill-3B} |
| } |
| ``` |
| |
| --- |
| |
| # Author |
| |
| **Siddharth N.R.** |
| |
| **Pluto AI Research Lab** |
| |
| --- |
| |
| ## License |
| |
| Apache-2.0 |
