ZalzalaKhan/NLP-Project-2

Qwen2.5-0.5B-Instruct — GSM8K Math Reasoning (SFT → GRPO)

LoRA adapter fine-tuned on Qwen2.5-0.5B-Instruct for mathematical reasoning using a two-stage SFT → GRPO pipeline. Trained on Kaggle T4 GPU.

IBA Karachi · NLP with Deep Learning · Assignment 04 · Option C
Authors: Immaduddin Durrani, Raahin Tajuddin, Ibad Khan

Results

Stage	Judge Score	vs Baseline
Baseline (no fine-tuning)	5.93/10	—
Best SFT (T1)	6.40/10	+7.9%
Best GRPO (G1) ← this model	7.03/10	+18.6%

Evaluated on 30 held-out GSM8K test prompts using LLM-as-Judge (Llama 3.3-70B via NVIDIA NIM), 4-axis rubric (Correctness 0-4, Reasoning 0-3, Format 0-2, Conciseness 0-1).

How to Use

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("ZalzalaKhan/Qwen2.5-0.5B-GSM8K-GRPO")
tokenizer = AutoTokenizer.from_pretrained("ZalzalaKhan/Qwen2.5-0.5B-GSM8K-GRPO")

messages = [
    {"role": "system", "content": "Reason step by step and clearly provide the final numerical answer."},
    {"role": "user", "content": "Janet has 10 apples. She gives 3 to her friend. How many does she have left?"}
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
output = model.generate(input_ids, max_new_tokens=256, temperature=0.6, top_p=0.95)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Details

Pipeline Overview

Stage 1 — SFT (NB-1): Fine-tuned on MetaMathQA hard split (1,500 samples, LoRA rank-32, 2 epochs, 8 min on T4). Best trial T1 reached 6.40/10.

Stage 2 — GRPO (NB-2): Reinforcement learning from correctness reward on GSM8K train split (500 samples, group size 8, KL=0.1, 162 min on T4). Best trial G1 reached 7.03/10.

GRPO Hyperparameters (G1)

Parameter	Value
Base model	SFT T1 checkpoint
LoRA Rank	32
Target Modules	q_proj, v_proj
KL Coefficient	0.1
Learning Rate	1e-5
Group Size	8
Generation Temp	0.6
Train Samples	500 (GSM8K)
Training Time	162 min (Kaggle T4)

Data

• SFT: MetaMathQA hard split (no overlap with test set)
• GRPO: GSM8K train split, stratified 400 medium + 100 easy
• Eval: 30 held-out GSM8K test prompts (10 easy / 10 medium / 10 hard)

Framework Versions

• PEFT 0.18.1
• Unsloth 2026.5.8
• TRL 0.24.0
• Transformers 5.5.0
• PyTorch 2.10.0+cu118