---
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
- triton
- kernelbook
- code-generation
- self-distillation
- sdft
- text-generation
datasets:
- custom
language:
- en
pipeline_tag: text-generation
---

# Qwen2.5-Coder-7B KernelBook SDFT

**Self-Distillation Fine-Tuning (SDFT)** checkpoint of [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct), post-trained on the **KernelBook** Triton kernel dataset.

## Method

This model was trained with **SDFT** (self-distillation fine-tuning): the student sees the user prompt plus privileged reference context (the target Triton implementation) and learns to reproduce the reference completion via forced-completion distillation (cross-entropy + KL on completion tokens). Training used a custom `KernelBookSDFTTrainer` on top of `transformers.Trainer` with DeepSpeed ZeRO-3.

## Dataset

- **KernelBook** — PyTorch module prompts paired with reference Triton kernels
- Deduplicated, filtered to completions ≤4096 tokens, repo-stratified 80/10/10 split
- **1 training epoch** on the KernelBook train split

## Intended use

Generate Triton GPU kernels from PyTorch-style module descriptions. Best for KernelBook-style conversion prompts; not evaluated as a general-purpose chat or reasoning model.

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "aadityabuilds/qwen2-5-coder-7b-kernelbook-sdft"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True
)

messages = [
    {
        "role": "user",
        "content": "Convert the following PyTorch code to an equivalent Triton kernel...",
    }
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1] :], skip_special_tokens=True))
```

## Training summary

| Setting | Value |
|---------|-------|
| Base model | Qwen2.5-Coder-7B-Instruct |
| Method | SDFT (forced-completion distillation) |
| Epochs | 1 |
| Hardware | 4× H100 (Modal) |
| Parallelism | DeepSpeed ZeRO-3, bf16 |

## Limitations

Specialized for KernelBook Triton codegen. May show reduced performance on general coding, math, and knowledge benchmarks compared to the base instruct model.