--- license: apache-2.0 language: - en base_model: - unsloth/Qwen2.5-3B-Instruct-unsloth-bnb-4bit library_name: adapter-transformers tags: - text-generation-inference - transformers - unsloth - qwen2 - trl - grpo --- # Uploaded model - **Developed by:** Akshint47 - **License:** apache-2.0 - **Finetuned from model :** unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit This qwen2 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. [](https://github.com/unslothai/unsloth) # Fine-Tuning Qwen2.5-3B-Instruct with GRPO for GSM8K Dataset This notebook demonstrates the process of fine-tuning the **Qwen2.5-3B-Instruct** model using **GRPO (Generalized Reward Policy Optimization)** on the **GSM8K** dataset. The goal is to improve the model's ability to solve mathematical reasoning problems by leveraging reinforcement learning with custom reward functions. ## Overview The notebook is structured as follows: 1. **Installation**: Installs necessary libraries such as `unsloth`, `vllm`, and `trl` for efficient fine-tuning and inference. 2. **Unsloth Setup**: Configures the environment for faster fine-tuning using Unsloth's `PatchFastRL` and loads the Qwen2.5-3B-Instruct model with LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning. 3. **Data Preparation**: Loads and preprocesses the GSM8K dataset, formatting it for training with a system prompt and XML-style reasoning and answer format. 4. **Reward Functions**: Defines custom reward functions to evaluate the model's responses, including: - **Correctness Reward**: Checks if the extracted answer matches the ground truth. - **Format Reward**: Ensures the response follows the specified XML format. - **Integer Reward**: Verifies if the extracted answer is an integer. - **XML Count Reward**: Evaluates the completeness of the XML structure in the response. 5. **GRPO Training**: Configures and runs the GRPO trainer with vLLM for fast inference, using the defined reward functions to optimize the model's performance. 6. **Training Progress**: Monitors the training progress, including rewards, completion length, and KL divergence, to ensure the model is improving over time. ## Key Features - **Efficient Fine-Tuning**: Utilizes Unsloth and LoRA to fine-tune the model with reduced memory usage and faster training times. - **Custom Reward Functions**: Implements multiple reward functions to guide the model towards generating correct and well-formatted responses. - **vLLM Integration**: Leverages vLLM for fast inference during training, enabling efficient generation of multiple responses for reward calculation. - **GSM8K Dataset**: Focuses on improving the model's performance on mathematical reasoning tasks, specifically the GSM8K dataset. ## Requirements - Python 3.11 - Libraries: `unsloth`, `vllm`, `trl`, `torch`, `transformers` ## Installation To set up the environment, run: ```bash pip install unsloth vllm trl ``` ## Usage - **Load the Model**: The notebook loads the Qwen2.5-3B-Instruct model with LoRA for fine-tuning. - **Prepare the Dataset**: The GSM8K dataset is loaded and formatted with a system prompt and XML-style reasoning and answer format. - **Define Reward Functions**: Custom reward functions are defined to evaluate the model's responses. - **Train the Model**: The GRPO trainer is configured and run to fine-tune the model using the defined reward functions. - **Monitor Progress**: The training progress is monitored, including rewards, completion length, and KL divergence. ## Results - The training process is designed to improve the model's ability to generate correct and well-formatted responses to mathematical reasoning problems. The reward functions guide the model towards better performance, and the training progress is logged for analysis. ## Future Work - **Hyperparameter Tuning**: Experiment with different learning rates, batch sizes, and reward weights to optimize performance. - **Additional Datasets**: Extend the fine-tuning process to other datasets to improve the model's generalization capabilities. - **Advanced Reward Functions**: Implement more sophisticated reward functions to further refine the model's responses. ## Acknowledgments - **Unsloth**: For providing tools to speed up fine-tuning. - **vLLM**: For enabling fast inference during training. - **Hugging Face**: For the trl library and the GSM8K dataset. - Special thanks to @sudhir2016 sir for mentoring me for developing such a prominent fine-tuning model. ## License This project is licensed under the MIT License. See the LICENSE file for details.