File size: 1,622 Bytes
d6586df f37a410 16b8074 f37a410 16b8074 f37a410 16b8074 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
{}
---
# CodeGPT Fine-tuned for Code Generation
## Model Description
This model is a fine-tuned version of [microsoft/CodeGPT-small-py](https://huggingface.co/microsoft/CodeGPT-small-py)
trained on coding problems and solutions for code generation tasks.
## Training Details
- **Base Model:** microsoft/CodeGPT-small-py (124M parameters)
- **Dataset:** Rabinovich/Code-Generation-LLM-LoRA (500 examples)
- **Epochs:** 2
- **Learning Rate:** 5e-5
- **Batch Size:** 4
- **Hardware:** CPU
## Training Results
| Step | Training Loss |
|------|--------------|
| 25 | 4.4322 |
| 50 | 3.4648 |
| 100 | 3.1430 |
| 150 | 2.7050 |
| 200 | 2.7491 |
| 250 | 2.7126 |
**Loss improved from 4.43 → 2.71 (39% reduction)**
## How to Use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Pradnya27/codegpt-finetuned-code-generation")
tokenizer = AutoTokenizer.from_pretrained("Pradnya27/codegpt-finetuned-code-generation")
prompt = "Generate code: Write a function to check if a number is prime"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Limitations
- Trained on a small subset (500 examples) — larger training would improve results
- Works best with competitive programming style problems
- Output quality improves with more specific prompts
## Future Work
- Train on full dataset (34,727 examples)
- Experiment with LoRA fine-tuning
- Evaluate on HumanEval benchmark
|