qwen2.5-3b-lora-r24

Fine-tuned LoRA adapter on top of Qwen/Qwen2.5-3B-Instruct.

Training Details

Base Model

• Model: Qwen/Qwen2.5-3B-Instruct

LoRA Configuration

Parameter	Value
r	24
alpha	48
dropout	0.05
target modules	down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
task type	CAUSAL_LM

Training Arguments

Parameter	Value
epochs	3.0
batch size	2
learning rate	2e-05
weight decay	0.001
warmup steps	2.0
gradient accumulation steps	1

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "qwen2.5-3b-lora-r24")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")