---
library_name: transformers
tags: []
---

# MNLP M3 DPO Model — Qwen3-0.6B-Base Fine-Tuned with Direct Preference Optimization

This repository contains a Direct Preference Optimization (DPO) model built on top of the base model [`Qwen/Qwen3-0.6B-Base`](https://huggingface.co/Qwen/Qwen3-0.6B-Base), as part of the MNLP M3 project. The model is fine-tuned using a high-quality preference dataset to better align responses with human preferences.

## Model Description

- **Base Model**: [`Qwen/Qwen3-0.6B-Base`](https://huggingface.co/Qwen/Qwen3-0.6B-Base)
- **DPO Dataset**: [`Tandogan/MNLP_M3_dpo_dataset`](https://huggingface.co/datasets/Tandogan/MNLP_M3_dpo_dataset)
- **Libraries**: [Unsloth](https://github.com/unslothai/unsloth), [TRL](https://github.com/huggingface/trl)

## Training Procedure

### Direct Preference Optimization (DPO)

We started from the official `Qwen3-0.6B-Base` checkpoint and applied **Direct Preference Optimization (DPO)**.  
DPO lets us directly train the model to score preferred responses higher than less-preferred ones, using ranked human feedback.  

#### 1. From Base Model (`Qwen3-0.6B-Base`)

- **Dataset**: [`Tandogan/MNLP_M3_dpo_dataset`](https://huggingface.co/datasets/Tandogan/MNLP_M3_dpo_dataset)
- **Max sequence length**: 2048 (prompt truncated to 1024)
- **Epochs**: 4  
- **Optimizer**: AdamW (learning rate = `2e-6`, weight decay = `0`)  
- **Precision**: bf16  
- **Batch size**: 2 (gradient accumulation = 4)  
- **Scheduler**: cosine with 1% warmup  
- **DPO Beta**: 0.1  
- **Eval & Checkpointing**: Every epoch  
- **Monitoring**: Weights & Biases (WandB)  
- **Best Epoch Selection**: Based on reward accuracy
## Evaluation

| Model                          | BLEU   | ROUGE-1/2/L/Lsum                      | METEOR | MMLU ± SD         | TQA_MC1 ± SD     | TQA_MC2 ± SD     |Reward Acc. ± SD    |
|--------------------------------|--------|---------------------------------------|--------|-------------------|------------------|------------------|--------------------|
| **Qwen3-0.6B-Base**            | 0.1086 | 0.3282 / 0.1458 / 0.2187 / 0.2964     | 0.2406 | 0.5239 ± 0.0365   | 0.2938 ± 0.0159  | 0.4589 ± 0.0148  |0 ± 0               |
| **Qwen3-0.6B**                 | 0.0649 | 0.2488 / 0.0876 / 0.1617 / 0.2224     | 0.2146 | 0.4156 ± 0.0361  | 0.2717 ± 0.0156 | 0.4284 ± 0.0145 | 0.4226 ± 0.0088       |
| **MNLP M3 DPO Model**          | 0.1343 | 0.3608 / 0.1634 / 0.2345 / 0.3283     | 0.2718 | 0.5264 ± 0.0364   | 0.3023 ± 0.0161  | 0.4682 ± 0.0149  |0.6997 ± 0.0082     |


## Intended Use

This model is intended for research and experimentation with preference-based alignment and reward modeling.

## How to Use

You can use the model with the `transformers` and `trl` libraries for inference or evaluation:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Tandogan/MNLP_M3_dpo_model").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("Tandogan/MNLP_M3_dpo_model")

prompt = "Explain recursion in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))