File size: 2,573 Bytes
238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b 238a8e5 8f19b4b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
library_name: transformers
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
- triton
- kernelbook
- code-generation
- self-distillation
- sdft
- text-generation
datasets:
- custom
language:
- en
pipeline_tag: text-generation
---
# Qwen2.5-Coder-7B KernelBook SDFT
**Self-Distillation Fine-Tuning (SDFT)** checkpoint of [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct), post-trained on the **KernelBook** Triton kernel dataset.
## Method
This model was trained with **SDFT** (self-distillation fine-tuning): the student sees the user prompt plus privileged reference context (the target Triton implementation) and learns to reproduce the reference completion via forced-completion distillation (cross-entropy + KL on completion tokens). Training used a custom `KernelBookSDFTTrainer` on top of `transformers.Trainer` with DeepSpeed ZeRO-3.
## Dataset
- **KernelBook** — PyTorch module prompts paired with reference Triton kernels
- Deduplicated, filtered to completions ≤4096 tokens, repo-stratified 80/10/10 split
- **1 training epoch** on the KernelBook train split
## Intended use
Generate Triton GPU kernels from PyTorch-style module descriptions. Best for KernelBook-style conversion prompts; not evaluated as a general-purpose chat or reasoning model.
## Quick start
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aadityabuilds/qwen2-5-coder-7b-kernelbook-sdft"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
messages = [
{
"role": "user",
"content": "Convert the following PyTorch code to an equivalent Triton kernel...",
}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1] :], skip_special_tokens=True))
```
## Training summary
| Setting | Value |
|---------|-------|
| Base model | Qwen2.5-Coder-7B-Instruct |
| Method | SDFT (forced-completion distillation) |
| Epochs | 1 |
| Hardware | 4× H100 (Modal) |
| Parallelism | DeepSpeed ZeRO-3, bf16 |
## Limitations
Specialized for KernelBook Triton codegen. May show reduced performance on general coding, math, and knowledge benchmarks compared to the base instruct model.
|