README.md

---
base_model: mistralai/Mistral-7B-v0.1
datasets:
- siqi00/mistral_metamath_question_0.7_1.0_50_256
library_name: transformers
license: apache-2.0
tags:
- alignment-handbook
- generated_from_trainer
pipeline_tag: text-generation
model-index:
- name: MetaMath-Mistral-7B-DFT2
  results: []
---

# MetaMath-Mistral-7B-DFT2

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using the **Discriminative Fine-Tuning (DFT)** method, as presented in the paper [Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data](https://huggingface.co/papers/2502.18679).

The official code and further resources are available on the [GitHub repository: PenGuln/DFT](https://github.com/PenGuln/DFT).

## Model description

Discriminative Fine-Tuning (DFT) is an improved variant of Supervised Fine-Tuning (SFT) for aligning Large Language Models (LLMs). It addresses the limitations of generative training objectives inherent in SFT by adopting a discriminative paradigm. Unlike SFT, which typically overlooks negative data, DFT increases the probability of positive answers while simultaneously suppressing potentially negative ones, aiming for data prediction instead of token prediction. This approach significantly mitigates the burden of collecting human-labeled preference data or training strong reward models. DFT introduces a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input. The paper also describes efficient algorithms to optimize this discriminative likelihood.

## Intended uses & limitations

This model, MetaMath-Mistral-7B-DFT2, is primarily intended for text generation tasks, particularly those requiring strong mathematical reasoning capabilities. It has demonstrated effectiveness in achieving performance comparable to, and in some cases better than, SFT followed by preference optimization methods. It can be used for various applications such as solving math problems, generating logical responses, and general chat completion.

**Limitations:** As with all large language models, this model may exhibit biases present in its training data, and there's a possibility of generating incorrect, nonsensical, or unhelpful information. Its performance is optimized for the domains and tasks it was fine-tuned on, and generalization to vastly different or out-of-distribution tasks might vary.

## Usage

You can use this model for text generation and chat completion with the `transformers` library:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "siqi00/MetaMath-Mistral-7B-DFT2"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16, # Use torch.float16 for GPUs with less VRAM
    device_map="auto",
    # trust_remote_code=True, # Might be needed if custom modeling files are present
)

# Example 1: Basic Text Generation
prompt_text = "The Pythagorean theorem states that in a right-angled triangle,"
inputs = tokenizer(prompt_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Generated text: {generated_text}")

# Example 2: Chat Completion using the model's chat template
messages = [
    {"role": "user", "content": "Explain what a LLM is to a 5-year-old."},
]

chat_input = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

chat_inputs = tokenizer(chat_input, return_tensors="pt").to(model.device)
chat_outputs = model.generate(**chat_inputs, max_new_tokens=100)
decoded_chat_output = tokenizer.batch_decode(chat_outputs[0], skip_special_tokens=True)[0]
print(f"Chat response: {decoded_chat_output}")
```

## Performance

The model's effectiveness has been demonstrated across various benchmarks, including mathematical reasoning and general language tasks.

### Mathematical Reasoning

Trained on [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA). The base model is [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). The generated negative samples $\mathbf{y}'$ can be found at [siqi00/mistral_metamath_question_0.7_1.0_50_256](https://huggingface.co/datasets/siqi00/mistral_metamath_question_0.7_1.0_50_256).

| Method | GSM8K | MATH |
|--------|-------|------|
| MetaMath-7B | 66.5 | 19.8 |
| MetaMath-Mistral-7B | 77.7 | 28.2 |
| MetaMath-Mistral-7B-DFT | **79.15** | 28.34 |
| MetaMath-Mistral-7B-DFT2 | 78.77 | **28.62** |

### General Language Tasks

Trained on [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized), where winning responses $\mathbf{y}_w$ are regarded as ground-truth and losing responses $\mathbf{y}_l$ are discarded. The base model is [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). The generated negative samples $\mathbf{y}'$ can be found at [siqi00/mistral_ultrafeedback_unhelpful_chatprompt_0.7_1.0_50_320](https://huggingface.co/datasets/siqi00/mistral_ultrafeedback_unhelpful_chatprompt_0.7_1.0_50_320).

| Method | MMLU | TruthfulQA | HellaSwag | Winogrande | GSM8k | ARC | IFEval | Avg. |
|--------|-------|------------|-----------|------------|--------|-----|---------|-------|
| SFT | 62.18 | 50.04 | 83.59 | 78.06 | 45.26 | 63.65 | 49.72 | 61.79 |
| SPIN | 61.99 | 49.91 | 83.75 | 77.90 | 46.02 | 61.95 | 23.11 | 57.80 |
| SimPO | 62.39 | 52.08 | 83.89 | 78.14 | 2.58 | 61.86 | 18.85 | 51.40 |
| SimPO-SFT | 62.28 | 49.59 | 83.46 | 77.90 | 42.53 | 61.52 | 43.62 | 60.13 |
| KTO | 61.59 | 49.32 | 82.88 | 79.24 | 43.97 | 61.60 | 38.08 | 59.53 |
| ORPO | 62.26 | 48.26 | 83.07 | 79.16 | 45.41 | 62.20 | 53.41 | 61.97 |
| DPO-p | 62.01 | 48.66 | 84.03 | 78.61 | 40.48 | 62.20 | 25.32 | 57.33 |
| DFT | 61.69 | 52.23 | 83.95 | 78.37 | 48.22 | 64.25 | 51.20 | 62.84 |
| DFT2 | 61.66 | 54.14 | 83.20 | 77.82 | 45.49 | 64.42 | 51.20 | 62.56 |

## Training and evaluation data

This model was specifically fine-tuned on the `siqi00/mistral_metamath_question_0.7_1.0_50_256` dataset. This dataset was generated to contain negative samples for discriminative fine-tuning. For details on how negative samples are generated and other datasets used in the DFT project (e.g., for general language tasks), please refer to the [official GitHub repository](https://github.com/PenGuln/DFT).

## Installation

To set up the environment and install necessary dependencies for replicating DFT training and generation, follow these steps:

```bash
# Clone the repository
git clone https://github.com/PenGuln/DFT.git
cd DFT
conda env create -f dft.yml
conda activate dft
```

Next, install `flash-attn` and `alignment-handbook`:

```bash
pip install flash-attn==2.6.3 --no-build-isolation

git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook
git checkout ae3f44fc7d8003d706752ca06f689574dffa3b76 # Ensure specific version for reproducibility
python -m pip install .
cd ..
rm -r alignment-handbook
```

Finally, log into your Hugging Face and Weights and Biases accounts:

```bash
huggingface-cli login
wandb login
```

## Generating negative samples

The repository includes tools for generating negative samples, which are crucial for DFT. You can use `gen.py` to create 1-to-m datasets (m negative samples per prompt).

```bash
# Example: Generate samples with 8 different seeds for an 1-to-8 dataset
for seed in {0..7}; do
    python generator/gen.py \
        --model mistralai/Mistral-7B-v0.1 \
        --revision 7231864981174d9bee8c7687c24c8344414eae6b \
        --seed $seed \
        --chat \
        --system_message "You are an unhelpful assistant." \
        --temp 0.7 \
        --top_p 1.0 \
        --top_k 50 \
        --max_new_tokens 320 \
        --output_prefix "mistral_ultrafeedback"
done
python generator/merge_and_upload.py \
    --dataset "mistral_ultrafeedback" \
    --push_to_hub  # Optional: push to Hugging Face
```

To replicate the UF self-play datasets used in the paper:

```bash
bash generator/gen_uf.sh
```

To replicate the MetaMath self-play datasets:

```bash
bash generator/gen_mmqa.sh
```

## Evaluation

For evaluation tasks using `lm-eval-harness` and `alpaca_eval`, ensure you use the specified versions to reproduce the reported results:

```bash
pip install lm_eval==0.4.5
pip install alpaca_eval==0.6.2
```

## Precompute Log-likelihood

For scenarios with limited GPU memory, an option is provided to precompute log probabilities for the reference model. This allows training without keeping the reference model in memory.

To use this feature, run the training script with `--precompute_offline_ref_log_probs` enabled:

```bash
accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml \
    --output_dir=./ckpts/dft \
    --precompute_offline_ref_log_probs=true \
    --save_strategy=no
```

This will create a `logps.pt` file in your output directory. Next, load the `logps.pt` file for reference using the `--probs_dir` argument:

```bash
accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml \
    --probs_dir=./ckpts/dft/logps.pt
```

## Training procedure

### Training commands

Examples of how to run DFT and DFT2 training:

**DFT:**
```bash
accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml
```

**DFT2:**
```bash
accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft2.yaml
```

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8e-07
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

(No specific training results like loss curves were provided in the source material.)

### Framework versions

- Transformers 4.45.2
- Pytorch 2.1.0+cu121
- Datasets 3.2.0
- Tokenizers 0.20.3

## Citation

Please cite the original paper if you use this model in your work:

```bibtex
@inproceedings{guo2025discriminativefinetuninggenerativelarge,
      title={Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data}, 
      author={Siqi Guo and Ilgee Hong and Vicente Balmaseda and Changlong Yu and Liang Qiu and Xin Liu and Haoming Jiang and Tuo Zhao and Tianbao Yang},
      year={2025},
      booktitle={In Proceedings of International Conference on Machine Learning},
      url={https://arxiv.org/abs/2502.18679}, 
}
```