thodsapon/GRPO_unsloth_Qwen2.5_3B

Model Card for Qwen/Qwen2.5-3B-Instruct (Finetuned with GRPO)

This model is a fine-tuned version of Qwen/Qwen2.5-3B-Instruct using the Unsloth framework and Generalized Reinforcement Policy Optimization (GRPO) from the trl library. The model has been optimized for instruction-following and reasoning tasks using LoRA with a rank of 64, aiming for efficient adaptation with minimal hardware requirements.

Code

https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(3B)-GRPO.ipynb#scrollTo=cXk993X6C2ZZ

Model Details

Model Description

• Developed by: [Your Name or Organization]
• Shared by: [Your Hugging Face username]
• Model type: Decoder-only transformer (LLM)
• Language(s): English, multilingual support inherited from base model
• License: Apache 2.0 (same as base model)
• Fine-tuned from model: Qwen/Qwen2.5-3B-Instruct

This model leverages the Unsloth ecosystem to support efficient training with LoRA adapters in 4-bit precision. GRPO was used as the fine-tuning strategy to reward structured reasoning in completions.

Model Sources

• Repository: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
• Frameworks: Unsloth, TRL

Uses

Direct Use

This model is suitable for:

• Instruction-following tasks
• Question answering
• Chain-of-thought reasoning
• Conversational AI
• Few-shot NLP tasks

Out-of-Scope Use

This model is not suitable for:

• Legal, medical, or financial advice without expert review
• Real-time safety-critical decision-making
• Generating harmful, biased, or toxic content

Bias, Risks, and Limitations

This model inherits any biases and limitations from its base model (Qwen2.5-3B-Instruct). As with all large language models, it may produce factually incorrect or misleading information.

Recommendations

• Apply content moderation and safety filters before deploying publicly.
• Human review is recommended in high-stakes use cases.

How to Get Started with the Model

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen2.5-3B-Instruct",
    max_seq_length = 1024,
    load_in_4bit = True,
    fast_inference = True,
    max_lora_rank = 64,
    gpu_memory_utilization = 0.5,
)

Training Details

Training Data

Data used for fine-tuning is assumed to contain instruction-response format prompts with reasoning chains. [Specify your dataset if available.]

Training Procedure

• LoRA Rank: 64
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Gradient Checkpointing: Enabled (via Unsloth)
• Precision: Mixed (bfloat16 or fp16 depending on hardware support)

Hyperparameters

• Learning rate: 5e-6
• Optimizer: AdamW 8-bit
• Warmup ratio: 0.1
• Scheduler: Cosine
• Batch size: 1
• Gradient accumulation: 1
• Max steps: 250
• Max prompt length: 256
• Max completion length: 200

Evaluation

Evaluation was done manually and qualitatively based on response coherence and instruction-following accuracy. [Include metrics if quantitative results are available.]

Environmental Impact

• Hardware Type: Consumer GPU (e.g., RTX 4060 or similar)
• Precision: 4-bit inference, bf16 or fp16 training
• Training Duration: ~X hours (estimated)
• Compute Region: [Your cloud provider/region]

Technical Specifications

• Model Architecture: Transformer decoder (Qwen)
• Training Framework: Unsloth, TRL, Hugging Face
• LoRA Adaptation: Enabled via Unsloth

Citation

If you use this model, please cite the base model and GRPO:

@misc{qwen25instruct,
  title={Qwen2.5-3B-Instruct},
  author={Qwen Team},
  year={2024},
  url={https://huggingface.co/Qwen/Qwen2.5-3B-Instruct}
}

@misc{grpo2025,
  title={Generalized Reinforcement Policy Optimization (GRPO)},
  author={TRL Team},
  year={2025},
  url={https://github.com/huggingface/trl}
}

Model Card Authors

[Leng Thodsapon]

Contact

---