README.md

---
license: mit
library_name: peft
pipeline_tag: text-generation
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
tags:
  - lora
  - sft
  - dpo
  - math
  - reasoning
  - llama-factory
  - deepseek-r1
  - qwen
datasets:
  - AI-MO/NuminaMath-CoT
language:
  - en
  - zh
---

# DeepMath (LoRA SFT + DPO on DeepSeek-R1-Distill-Qwen-7B)

> 🇨🇳 [中文简介](#中文) below.

**DeepMath** is a math-reasoning fine-tune of [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B), produced by a **two-stage LoRA pipeline** with **LLaMA-Factory** (template `deepseekr1`):

```
DeepSeek-R1-Distill-Qwen-7B ──SFT (LoRA, NuminaMath-CoT)──► merged ──DPO (LoRA, math prefs)──► DeepMath
```

This repo contains **LoRA adapters**, not a merged model. It was built in a 5-day sprint on rented consumer GPUs (8×5090 for SFT, 1×5090 for DPO) — the full engineering story is in the [GitHub repo](https://github.com/SoFarSoGoodya/DeepMath).

## ⚠️ What's in here & how to use it

There are **two adapters**, applied **in sequence**:

| Path | Role | Size |
|---|---|---|
| `sft_adapter/` | Stage 1 — SFT LoRA (NuminaMath-CoT) | 78 MB |
| `./` (top level) | Stage 2 — DPO LoRA (math preference pairs), the **final** adapter | 77 MB |

The DPO adapter was trained on top of the *SFT-merged* model, so to reconstruct the final model you must apply **SFT first, then DPO**. Use the included `merge.py`:

```bash
pip install transformers peft torch accelerate
python merge.py \
  --base deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --sft  ./sft_adapter \
  --dpo  ./ \
  --out  ./DeepMath-merged
```

Or load adapters on the fly (see `merge.py` docstring / the GitHub `examples/inference.py`).

## Training summary (honest)

| Stage | Config | Outcome |
|---|---|---|
| **SFT** | 8×5090, native DDP, LoRA r=8/α=16 (target=all), lr 5e-5, 1 epoch, cutoff 2048, 1,410 steps | loss **0.73 → 0.39** |
| **DPO** | 1×5090, **QLoRA** 4-bit NF4 + double-quant, lr 1e-6, 3 epochs, cutoff 1024, 456 steps | loss **0.6947** (≈ ln 2 baseline), reward acc ~0.5–0.55 |

![SFT loss](sft_training_loss.png)
![DPO reward accuracy](dpo_rewards_accuracies.png)

## Limitations & intended use

- **This is a learning/pipeline-demonstration artifact, not a SOTA model.** SFT converged cleanly; DPO completed without mode collapse but its gain is **modest** (reward accuracy only slightly above the 50% baseline) and **no held-out benchmark was evaluated**.
- Best understood as a reproducible reference for a resource-constrained SFT→DPO pipeline. Don't deploy it expecting strong math-reasoning gains.
- Base model `DeepSeek-R1-Distill-Qwen-7B` inherits its biases/limits; tokenizer derives from Qwen.

## Provenance & license

- **Base**: [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) (MIT).
- **SFT data**: [`AI-MO/NuminaMath-CoT`](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) (Apache-2.0), cleaned 100k→90,217.
- **DPO data**: 2,418 math preference pairs (distilabel-math-preference style).
- **Adapters + code**: **MIT**. Please retain upstream attributions.

**Author**: [@SoFarSoGoodya](https://github.com/SoFarSoGoodya) — full attribution & contact in the [GitHub repo](https://github.com/SoFarSoGoodya/DeepMath).

---

## 中文

**DeepMath** 是基于 `DeepSeek-R1-Distill-Qwen-7B` 的数学推理微调,用 **LLaMA-Factory** 走 **LoRA SFT(NuminaMath)→ 合并 → LoRA DPO(数学偏好对)** 两段式流水线。

本仓库是 **LoRA adapter**(非合并模型)。含两个 adapter,需**按顺序**使用:先 `sft_adapter/`(SFT),再顶层目录(DPO,最终)。用自带的 `merge.py` 一条命令重建完整模型(见上方命令)。

**如实说明**:这是一个学习/流水线演示作品,**不是 SOTA 模型**。SFT 收敛良好;DPO 完整跑通但提升有限(奖励准确率仅略高于随机),且未做独立评测。权重与代码采用 **MIT**;请保留对上游(DeepSeek / Qwen / NuminaMath)的署名。完整工程复盘见 [GitHub 仓库](https://github.com/SoFarSoGoodya/DeepMath)。