README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- code
- python
- lora
- fine-tuning
- qwen
datasets:
- Naholav/CodeGen-Deep-5K
- Naholav/CodeGen-Diverse-5K
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
pipeline_tag: text-generation
---

# cazim/homework-qwen-coder-deep-lora

This model is a LoRA fine-tuned version of [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) for competitive code generation.

## Model Description

- **Base Model:** Qwen/Qwen2.5-Coder-1.5B-Instruct
- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
- **Dataset:** Naholav/CodeGen-Deep-5K
- **Task:** Competitive Code Generation

## Training Details

### LoRA Configuration

| Parameter | Value |
|-----------|-------|
| Rank (r) | 32 |
| Alpha | 64 |
| Dropout | 0.1 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |

### Training Configuration

| Parameter | Value |
|-----------|-------|
| Learning Rate | 0.0002 |
| Epochs | 3 |
| Batch Size | 1 |
| Gradient Accumulation | 16 |
| Effective Batch Size | 16 |
| Max Sequence Length | 768 |
| Warmup Ratio | 0.1 |
| Weight Decay | 0.01 |
| LR Scheduler | cosine |

## Usage

### Using the LoRA Adapter

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "cazim/homework-qwen-coder-deep-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct")

# Generate code
system_prompt = "You are an expert Python programmer. Please read the problem carefully before writing any Python code."
problem = "Write a function to find the factorial of a number."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": problem},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.1,
    top_p=0.95,
    do_sample=True,
)

generated_code = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(generated_code)
```

### Using Merged Model

If this is a merged model, you can use it directly:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "cazim/homework-qwen-coder-deep-lora",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("cazim/homework-qwen-coder-deep-lora")
```

## Evaluation Results

Evaluation results will be added after benchmark testing.

## Limitations

- This model is fine-tuned specifically for competitive programming problems
- Best used with the provided system prompt
- May not generalize well to other code generation tasks

## System Prompt

```
You are an expert Python programmer. Please read the problem carefully before writing any Python code.
```

## Citation

If you use this model, please cite:

```bibtex
@misc{cazim_homework_qwen_coder_deep_lora},
  title={LoRA Fine-tuned Qwen2.5-Coder for Code Generation},
  year={2024},
  publisher={HuggingFace},
}
```

## License

This model is released under the Apache 2.0 license.