tensorov/qwen2.5-coder-7b-dpo

Qwen2.5-Coder-7B-Instruct — DPO (Combined Reasoning)

This is a LoRA adapter for unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit, fine-tuned with a two-stage pipeline (SFT → DPO) on the Combined Reasoning Distill dataset — a collection of 1.33M reasoning traces distilled from frontier models (Claude Opus 4.6/4.7, GPT 5.1/5.2, Kimi K2.5/K2.6, GLM 5.1, Gemini 3 Pro, MiniMax M2.1, and others).

The adapter is loaded on top of the 4-bit quantized base model, making it suitable for inference on GPUs with as little as 8 GB VRAM.

Model Details

• Base model: Qwen2.5-Coder-7B-Instruct (4-bit quantized via unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit)
• Architecture: Qwen2ForCausalLM, 7B parameters (80.7M LoRA trainable)
• Adapter type: LoRA (rank=32, alpha=32, dropout=0)
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training stages: SFT → DPO
• License: Apache 2.0
• Language: English

Training Data

The Combined Reasoning Distill dataset consists of 1,335,511 records aggregated from 41 source datasets covering reasoning traces from:

Source Family	Models
Anthropic	Claude Opus 4.5/4.6/4.7, Sonnet 4.5/4.6, Haiku 4.5
OpenAI	GPT 5.1/5.2
Kimi	K2, K2.5, K2.6
GLM	4.6, 4.7, 5.1
Google	Gemini 3 Pro Preview
MiniMax	M2.1
xAI	Grok Code Fast 1

The data covers math, code, science, logic, and general reasoning. Thinking traces are embedded in <think>...</think> tags where present.

Training splits

Split	SFT stage	DPO stage
Train	439,942 records	420,111 preference pairs
Eval	13,607 records	12,996 preference pairs

Training Procedure

Stage 1 — Supervised Fine-Tuning (SFT)

The model was first fine-tuned on the best-ranked answer per question group (think tags removed).

Hyperparameter	Value
Learning rate	2e-5
Schedule	Linear with 5% cosine warmup
Batch size	16 (gradient accumulation)
Epochs	1
Max seq length	4,096 tokens
Optimizer	AdamW (β₁=0.9, β₂=0.999)
Weight decay	0.01
LoRA rank	64 (SFT stage)
LoRA alpha	128
Dropout	0.05
Steps	27,497 (best eval loss checkpoint)

Stage 2 — Direct Preference Optimization (DPO)

The SFT checkpoint was used as the base for DPO training on preference pairs (chosen/rejected from answer rankings).

Hyperparameter	Value
Learning rate	1e-4
Schedule	Linear with 10% cosine warmup
Batch size	16 (effective, accum=16 × batch=1)
Epochs	1
Max seq length	1,024 tokens
β (DPO temperature)	0.1
Optimizer	AdamW 8-bit
Weight decay	0.01
LoRA rank	32 (DPO stage)
LoRA alpha	32
Dropout	0
Steps	8,000
Training time	~105 hours
Final loss	0.1403

Hardware

• GPUs: 2× Tesla V100-SXM2 16 GB (no NVLink)
• CUDA: 13.2
• PyTorch: 2.10.0+cu128
• Framework: Unsloth (SFT) + TRL DPOTrainer (DPO)
• Precision: fp16 mixed precision (V100 has no bf16 support)
• Memory: ~9.6 GiB / ~7.6 GiB (GPU 0 / GPU 1, model split across both via device_map="auto")

Environmental Impact

• Hardware: 2× Tesla V100 16 GB
• Training time: ~105 hours (DPO) + ~30 hours (SFT) = ~135 hours total
• Power: ~300W per GPU × 2 GPUs × 135h ≈ 81 kWh
• CO₂ equivalent: ~35–55 kg CO₂ (varies by grid mix)

Evaluation Results

Benchmark	Score	Notes
GSM8K (CoT, 0-shot)	30.0% (15/50)	Unsloth-based eval
MMLU (0-shot, 100 samples)	48.0% (48/100)	PEFT eval
HumanEval (pass@50)	100.0% (50/50)	PEFT eval, name-presence heuristic

Notes on Evaluation

• GSM8K was evaluated with chain-of-thought prompting (0-shot).
• MMLU was evaluated 0-shot on 100 samples covering STEM, humanities, and social sciences.
• HumanEval pass@50 uses a name-presence heuristic (checks if the expected function name appears in generated code); this likely overestimates true pass@k. Exact match or test-case-based evaluation would yield a lower score.
• Benchmarks were run using PEFT (for MMLU and HumanEval) due to a torch.compile incompatibility in the Unsloth inference path. GSM8K was evaluated via Unsloth with a patched environment.

How to Use

Load the adapter

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Base model (4-bit quantized)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit",
    device_map="auto",
    torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit")

# Load DPO adapter
model = PeftModel.from_pretrained(base_model, "tensorov/qwen2.5-coder-7b-dpo")
model.eval()

Inference example

prompt = "Write a Python function to check if a string is a palindrome, ignoring case and spaces."
messages = [
    {"role": "user", "content": prompt}
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, do_sample=True
    )
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Loading with Unsloth (if available)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "tensorov/qwen2.5-coder-7b-dpo",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Limitations

• Benchmark coverage is limited. Only GSM8K (50 samples), MMLU (100 samples), and HumanEval were evaluated. Broader evaluation (MATH, MBPP, BigBench, etc.) was not performed.
• HumanEval 100% is inflated. The evaluation used a name-presence heuristic, not actual test-case execution.
• DPO training used seq=1024. Longer-context reasoning (>2K tokens) was not explicitly trained, though the base model supports up to 131K tokens.
• Single-epoch DPO. Training was capped at 1 epoch based on the recommendation in the DPO literature; additional epochs could potentially improve alignment at the cost of overfitting risk.
• 4-bit quantization. The base model uses 4-bit NormalFloat quantization, which slightly degrades output quality compared to fp16 inference.
• English only. The model was trained exclusively on English data.

Citation

If you use this adapter, please cite the base model and the training dataset:

@article{qwen2.5-coder,
  title={Qwen2.5-Coder Technical Report},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2024}
}

@misc{combined-reasoning-distill,
  title={Combined Reasoning Distill},
  author={Tensorov},
  year={2025},
  url={https://huggingface.co/tensorov/qwen2.5-coder-7b-dpo}
}

Repository Files

File	Size	Description
adapter_model.safetensors	323 MB	LoRA adapter weights
adapter_config.json	1.2 KB	LoRA configuration
tokenizer.json	11.4 MB	Tokenizer
tokenizer_config.json	4.6 KB	Tokenizer config
training_args.bin	6.3 KB	Training hyperparameters
chat_template.jinja	2.5 KB	Chat template
README.md	—	This file