---
language: en
license: mit
tags:
- code-generation
- python
- lora
- peft
- causal-lm
base_model: microsoft/CodeGPT-small-py
---

# CodeGPT LoRA Fine-tuned for Code Generation

Fine-tuned version of CodeGPT using LoRA (Low-Rank Adaptation) for Python code generation.

## 🔗 Links
- **Live Demo:** https://huggingface.co/spaces/Pradnya27/code-generator
- **Full Fine-tuned Model:** https://huggingface.co/Pradnya27/codegpt-finetuned-code-generation
- **GitHub:** https://github.com/pradnyagundu/codegpt-finetuned-code-generation

## Model Details
- **Base model:** microsoft/CodeGPT-small-py (124M parameters)
- **Method:** LoRA (Low-Rank Adaptation)
- **Trainable parameters:** 589,824 (0.36% of total)
- **Model size:** 2.36MB (vs 651MB for full fine-tuning)
- **Dataset:** Rabinovich/Code-Generation-LLM-LoRA (5000 examples)

## Training Details
- **Epochs:** 2
- **Learning rate:** 3e-4
- **Batch size:** 8
- **LoRA rank:** 16
- **LoRA alpha:** 32
- **Hardware:** Google Colab T4 GPU
- **Training time:** ~9 minutes

## Training Results
| Step | Loss |
|------|------|
| 100 | 4.28 |
| 300 | 3.45 |
| 500 | 3.28 |
| 700 | 3.24 |
| 900 | 3.15 |
| 1200 | 3.14 |

## Comparison vs Full Fine-tuning
| | Full Fine-tune | LoRA |
|---|---|---|
| Final loss | 2.31 | 3.14 |
| Model size | 651MB | 2.36MB |
| Training time | ~14 min | ~9 min |
| Trainable params | 124M | 589K |

## How to Use
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model = AutoModelForCausalLM.from_pretrained("microsoft/CodeGPT-small-py")
model = PeftModel.from_pretrained(base_model, "Pradnya27/codegpt-lora-code-generation")
tokenizer = AutoTokenizer.from_pretrained("microsoft/CodeGPT-small-py")
tokenizer.pad_token = tokenizer.eos_token

prompt = "Generate code: Write a function to check if a number is prime"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(inputs["input_ids"], max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Limitations
- Trained on competitive programming problems — works best for algorithmic tasks
- Small base model (124M params) limits output quality
- Full fine-tuning achieves lower loss on this dataset

## Future Work
- Train on full 34K dataset
- Increase LoRA rank to r=32 or r=64
- Evaluate on HumanEval benchmark