| --- |
| base_model: unsloth/Qwen2.5-0.5B-Instruct |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - base_model:adapter:unsloth/Qwen2.5-0.5B-Instruct |
| - grpo |
| - lora |
| - math |
| - gsm8k |
| - transformers |
| - trl |
| - unsloth |
| --- |
| |
| # Qwen2.5-0.5B-Instruct — GSM8K Math Reasoning (SFT → GRPO) |
|
|
| LoRA adapter fine-tuned on Qwen2.5-0.5B-Instruct for mathematical reasoning |
| using a two-stage SFT → GRPO pipeline. Trained on Kaggle T4 GPU. |
|
|
| **IBA Karachi · NLP with Deep Learning · Assignment 04 · Option C** |
| **Authors:** Immaduddin Durrani, Raahin Tajuddin, Ibad Khan |
|
|
| ## Results |
|
|
| | Stage | Judge Score | vs Baseline | |
| |-------|------------|-------------| |
| | Baseline (no fine-tuning) | 5.93/10 | — | |
| | Best SFT (T1) | 6.40/10 | +7.9% | |
| | **Best GRPO (G1) ← this model** | **7.03/10** | **+18.6%** | |
|
|
| Evaluated on 30 held-out GSM8K test prompts using LLM-as-Judge |
| (Llama 3.3-70B via NVIDIA NIM), 4-axis rubric |
| (Correctness 0-4, Reasoning 0-3, Format 0-2, Conciseness 0-1). |
|
|
| ## How to Use |
|
|
| ```python |
| from peft import AutoPeftModelForCausalLM |
| from transformers import AutoTokenizer |
| |
| model = AutoPeftModelForCausalLM.from_pretrained("ZalzalaKhan/Qwen2.5-0.5B-GSM8K-GRPO") |
| tokenizer = AutoTokenizer.from_pretrained("ZalzalaKhan/Qwen2.5-0.5B-GSM8K-GRPO") |
| |
| messages = [ |
| {"role": "system", "content": "Reason step by step and clearly provide the final numerical answer."}, |
| {"role": "user", "content": "Janet has 10 apples. She gives 3 to her friend. How many does she have left?"} |
| ] |
| |
| input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True) |
| output = model.generate(input_ids, max_new_tokens=256, temperature=0.6, top_p=0.95) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Training Details |
|
|
| ### Pipeline Overview |
|
|
| **Stage 1 — SFT (NB-1):** Fine-tuned on MetaMathQA hard split (1,500 samples, |
| LoRA rank-32, 2 epochs, 8 min on T4). Best trial T1 reached 6.40/10. |
|
|
| **Stage 2 — GRPO (NB-2):** Reinforcement learning from correctness reward on |
| GSM8K train split (500 samples, group size 8, KL=0.1, 162 min on T4). |
| Best trial G1 reached 7.03/10. |
|
|
| ### GRPO Hyperparameters (G1) |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Base model | SFT T1 checkpoint | |
| | LoRA Rank | 32 | |
| | Target Modules | q_proj, v_proj | |
| | KL Coefficient | 0.1 | |
| | Learning Rate | 1e-5 | |
| | Group Size | 8 | |
| | Generation Temp | 0.6 | |
| | Train Samples | 500 (GSM8K) | |
| | Training Time | 162 min (Kaggle T4) | |
|
|
| ### Data |
|
|
| - **SFT:** MetaMathQA hard split (no overlap with test set) |
| - **GRPO:** GSM8K train split, stratified 400 medium + 100 easy |
| - **Eval:** 30 held-out GSM8K test prompts (10 easy / 10 medium / 10 hard) |
|
|
| ## Framework Versions |
|
|
| - PEFT 0.18.1 |
| - Unsloth 2026.5.8 |
| - TRL 0.24.0 |
| - Transformers 5.5.0 |
| - PyTorch 2.10.0+cu118 |
