qwen2.5-3b-lora-r32

Fine-tuned LoRA adapter on top of Qwen/Qwen2.5-3B-Instruct.

Training Details

Base Model

• Model: Qwen/Qwen2.5-3B-Instruct

LoRA Configuration

Parameter	Value
r	32
alpha	64
dropout	0.05
target modules	down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
task type	CAUSAL_LM

Training Arguments

Parameter	Value
epochs	8.0
batch size	2
learning rate	0.0002
weight decay	0.0
warmup steps	2.0
gradient accumulation steps	1

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "qwen2.5-3b-lora-r32")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")