README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - code
  - python
  - lora
  - fine-tuning
  - qwen
datasets:
  - Naholav/CodeGen-Deep-5K
  - Naholav/CodeGen-Diverse-5K
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
pipeline_tag: text-generation

cazim/homework-qwen-coder-deep-lora

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-Coder-1.5B-Instruct for competitive code generation.

Model Description

• Base Model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Dataset: Naholav/CodeGen-Deep-5K
• Task: Competitive Code Generation

Training Details

LoRA Configuration

Parameter	Value
Rank (r)	32
Alpha	64
Dropout	0.1
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Configuration

Parameter	Value
Learning Rate	0.0002
Epochs	3
Batch Size	1
Gradient Accumulation	16
Effective Batch Size	16
Max Sequence Length	768
Warmup Ratio	0.1
Weight Decay	0.01
LR Scheduler	cosine

Usage

Using the LoRA Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "cazim/homework-qwen-coder-deep-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct")

# Generate code
system_prompt = "You are an expert Python programmer. Please read the problem carefully before writing any Python code."
problem = "Write a function to find the factorial of a number."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": problem},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.1,
    top_p=0.95,
    do_sample=True,
)

generated_code = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(generated_code)

Using Merged Model

If this is a merged model, you can use it directly:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "cazim/homework-qwen-coder-deep-lora",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("cazim/homework-qwen-coder-deep-lora")

Evaluation Results

Evaluation results will be added after benchmark testing.

Limitations

• This model is fine-tuned specifically for competitive programming problems
• Best used with the provided system prompt
• May not generalize well to other code generation tasks

System Prompt

You are an expert Python programmer. Please read the problem carefully before writing any Python code.

Citation

If you use this model, please cite:

@misc{cazim_homework_qwen_coder_deep_lora},
  title={LoRA Fine-tuned Qwen2.5-Coder for Code Generation},
  year={2024},
  publisher={HuggingFace},
}

License

This model is released under the Apache 2.0 license.