Improve model card: Add pipeline tag, paper/code links, usage, and detailed info

README.md

@@ -1,38 +1,226 @@
 ---
+base_model: mistralai/Mistral-7B-v0.1
+datasets:
+- siqi00/mistral_metamath_question_0.7_1.0_50_256
 library_name: transformers
 license: apache-2.0
-base_model: mistralai/Mistral-7B-v0.1
 tags:
 - alignment-handbook
 - generated_from_trainer
-datasets:
-- siqi00/mistral_metamath_question_0.7_1.0_50_256
+pipeline_tag: text-generation
 model-index:
 - name: MetaMath-Mistral-7B-DFT2
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # MetaMath-Mistral-7B-DFT2
 
-This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the siqi00/mistral_metamath_question_0.7_1.0_50_256 dataset.
+This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using the **Discriminative Fine-Tuning (DFT)** method, as presented in the paper [Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data](https://huggingface.co/papers/2502.18679).
+
+The official code and further resources are available on the [GitHub repository: PenGuln/DFT](https://github.com/PenGuln/DFT).
 
 ## Model description
 
-More information needed
+Discriminative Fine-Tuning (DFT) is an improved variant of Supervised Fine-Tuning (SFT) for aligning Large Language Models (LLMs). It addresses the limitations of generative training objectives inherent in SFT by adopting a discriminative paradigm. Unlike SFT, which typically overlooks negative data, DFT increases the probability of positive answers while simultaneously suppressing potentially negative ones, aiming for data prediction instead of token prediction. This approach significantly mitigates the burden of collecting human-labeled preference data or training strong reward models. DFT introduces a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input. The paper also describes efficient algorithms to optimize this discriminative likelihood.
 
 ## Intended uses & limitations
 
-More information needed
+This model, MetaMath-Mistral-7B-DFT2, is primarily intended for text generation tasks, particularly those requiring strong mathematical reasoning capabilities. It has demonstrated effectiveness in achieving performance comparable to, and in some cases better than, SFT followed by preference optimization methods. It can be used for various applications such as solving math problems, generating logical responses, and general chat completion.
+
+**Limitations:** As with all large language models, this model may exhibit biases present in its training data, and there's a possibility of generating incorrect, nonsensical, or unhelpful information. Its performance is optimized for the domains and tasks it was fine-tuned on, and generalization to vastly different or out-of-distribution tasks might vary.
+
+## Usage
+
+You can use this model for text generation and chat completion with the `transformers` library:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "siqi00/MetaMath-Mistral-7B-DFT2"
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16, # Use torch.float16 for GPUs with less VRAM
+    device_map="auto",
+    # trust_remote_code=True, # Might be needed if custom modeling files are present
+)
+
+# Example 1: Basic Text Generation
+prompt_text = "The Pythagorean theorem states that in a right-angled triangle,"
+inputs = tokenizer(prompt_text, return_tensors="pt").to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(f"Generated text: {generated_text}")
+
+# Example 2: Chat Completion using the model's chat template
+messages = [
+    {"role": "user", "content": "Explain what a LLM is to a 5-year-old."},
+]
+
+chat_input = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+chat_inputs = tokenizer(chat_input, return_tensors="pt").to(model.device)
+chat_outputs = model.generate(**chat_inputs, max_new_tokens=100)
+decoded_chat_output = tokenizer.batch_decode(chat_outputs[0], skip_special_tokens=True)[0]
+print(f"Chat response: {decoded_chat_output}")
+```
+
+## Performance
+
+The model's effectiveness has been demonstrated across various benchmarks, including mathematical reasoning and general language tasks.
+
+### Mathematical Reasoning
+
+Trained on [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA). The base model is [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). The generated negative samples $\mathbf{y}'$ can be found at [siqi00/mistral_metamath_question_0.7_1.0_50_256](https://huggingface.co/datasets/siqi00/mistral_metamath_question_0.7_1.0_50_256).
+
+| Method | GSM8K | MATH |
+|--------|-------|------|
+| MetaMath-7B | 66.5 | 19.8 |
+| MetaMath-Mistral-7B | 77.7 | 28.2 |
+| MetaMath-Mistral-7B-DFT | **79.15** | 28.34 |
+| MetaMath-Mistral-7B-DFT2 | 78.77 | **28.62** |
+
+### General Language Tasks
+
+Trained on [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized), where winning responses $\mathbf{y}_w$ are regarded as ground-truth and losing responses $\mathbf{y}_l$ are discarded. The base model is [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1). The generated negative samples $\mathbf{y}'$ can be found at [siqi00/mistral_ultrafeedback_unhelpful_chatprompt_0.7_1.0_50_320](https://huggingface.co/datasets/siqi00/mistral_ultrafeedback_unhelpful_chatprompt_0.7_1.0_50_320).
+
+| Method | MMLU | TruthfulQA | HellaSwag | Winogrande | GSM8k | ARC | IFEval | Avg. |
+|--------|-------|------------|-----------|------------|--------|-----|---------|-------|
+| SFT | 62.18 | 50.04 | 83.59 | 78.06 | 45.26 | 63.65 | 49.72 | 61.79 |
+| SPIN | 61.99 | 49.91 | 83.75 | 77.90 | 46.02 | 61.95 | 23.11 | 57.80 |
+| SimPO | 62.39 | 52.08 | 83.89 | 78.14 | 2.58 | 61.86 | 18.85 | 51.40 |
+| SimPO-SFT | 62.28 | 49.59 | 83.46 | 77.90 | 42.53 | 61.52 | 43.62 | 60.13 |
+| KTO | 61.59 | 49.32 | 82.88 | 79.24 | 43.97 | 61.60 | 38.08 | 59.53 |
+| ORPO | 62.26 | 48.26 | 83.07 | 79.16 | 45.41 | 62.20 | 53.41 | 61.97 |
+| DPO-p | 62.01 | 48.66 | 84.03 | 78.61 | 40.48 | 62.20 | 25.32 | 57.33 |
+| DFT | 61.69 | 52.23 | 83.95 | 78.37 | 48.22 | 64.25 | 51.20 | 62.84 |
+| DFT2 | 61.66 | 54.14 | 83.20 | 77.82 | 45.49 | 64.42 | 51.20 | 62.56 |
 
 ## Training and evaluation data
 
-More information needed
+This model was specifically fine-tuned on the `siqi00/mistral_metamath_question_0.7_1.0_50_256` dataset. This dataset was generated to contain negative samples for discriminative fine-tuning. For details on how negative samples are generated and other datasets used in the DFT project (e.g., for general language tasks), please refer to the [official GitHub repository](https://github.com/PenGuln/DFT).
+
+## Installation
+
+To set up the environment and install necessary dependencies for replicating DFT training and generation, follow these steps:
+
+```bash
+# Clone the repository
+git clone https://github.com/PenGuln/DFT.git
+cd DFT
+conda env create -f dft.yml
+conda activate dft
+```
+
+Next, install `flash-attn` and `alignment-handbook`:
+
+```bash
+pip install flash-attn==2.6.3 --no-build-isolation
+
+git clone https://github.com/huggingface/alignment-handbook.git
+cd alignment-handbook
+git checkout ae3f44fc7d8003d706752ca06f689574dffa3b76 # Ensure specific version for reproducibility
+python -m pip install .
+cd ..
+rm -r alignment-handbook
+```
+
+Finally, log into your Hugging Face and Weights and Biases accounts:
+
+```bash
+huggingface-cli login
+wandb login
+```
+
+## Generating negative samples
+
+The repository includes tools for generating negative samples, which are crucial for DFT. You can use `gen.py` to create 1-to-m datasets (m negative samples per prompt).
+
+```bash
+# Example: Generate samples with 8 different seeds for an 1-to-8 dataset
+for seed in {0..7}; do
+    python generator/gen.py \
+        --model mistralai/Mistral-7B-v0.1 \
+        --revision 7231864981174d9bee8c7687c24c8344414eae6b \
+        --seed $seed \
+        --chat \
+        --system_message "You are an unhelpful assistant." \
+        --temp 0.7 \
+        --top_p 1.0 \
+        --top_k 50 \
+        --max_new_tokens 320 \
+        --output_prefix "mistral_ultrafeedback"
+done
+python generator/merge_and_upload.py \
+    --dataset "mistral_ultrafeedback" \
+    --push_to_hub  # Optional: push to Hugging Face
+```
+
+To replicate the UF self-play datasets used in the paper:
+
+```bash
+bash generator/gen_uf.sh
+```
+
+To replicate the MetaMath self-play datasets:
+
+```bash
+bash generator/gen_mmqa.sh
+```
+
+## Evaluation
+
+For evaluation tasks using `lm-eval-harness` and `alpaca_eval`, ensure you use the specified versions to reproduce the reported results:
+
+```bash
+pip install lm_eval==0.4.5
+pip install alpaca_eval==0.6.2
+```
+
+## Precompute Log-likelihood
+
+For scenarios with limited GPU memory, an option is provided to precompute log probabilities for the reference model. This allows training without keeping the reference model in memory.
+
+To use this feature, run the training script with `--precompute_offline_ref_log_probs` enabled:
+
+```bash
+accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml \
+    --output_dir=./ckpts/dft \
+    --precompute_offline_ref_log_probs=true \
+    --save_strategy=no
+```
+
+This will create a `logps.pt` file in your output directory. Next, load the `logps.pt` file for reference using the `--probs_dir` argument:
+
+```bash
+accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml \
+    --probs_dir=./ckpts/dft/logps.pt
+```
 
 ## Training procedure
 
+### Training commands
+
+Examples of how to run DFT and DFT2 training:
+
+**DFT:**
+```bash
+accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft.yaml
+```
+
+**DFT2:**
+```bash
+accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_dft.py recipes/dft/mistral_base_dft2.yaml
+```
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -52,7 +240,7 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-
+(No specific training results like loss curves were provided in the source material.)
 
 ### Framework versions
 
@@ -60,3 +248,17 @@ The following hyperparameters were used during training:
 - Pytorch 2.1.0+cu121
 - Datasets 3.2.0
 - Tokenizers 0.20.3
+
+## Citation
+
+Please cite the original paper if you use this model in your work:
+
+```bibtex
+@inproceedings{guo2025discriminativefinetuninggenerativelarge,
+      title={Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data}, 
+      author={Siqi Guo and Ilgee Hong and Vicente Balmaseda and Changlong Yu and Liang Qiu and Xin Liu and Haoming Jiang and Tuo Zhao and Tianbao Yang},
+      year={2025},
+      booktitle={In Proceedings of International Conference on Machine Learning},
+      url={https://arxiv.org/abs/2502.18679}, 
+}
+```