---
language: en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- gpt2
- code-completion
- pytorch
- huggingface
- transformers
- codexglue
- nlp
- machine-learning
---


# GPT-2 Fine-Tuned for Python Code Completion

This repository contains a fine-tuned **GPT-2** model for **Python source code completion**. The model was trained on the **CodeXGLUE Python Code Completion** dataset using the Hugging Face Transformers library and PyTorch.

## Model Description

This model is designed to predict the next tokens in Python source code, enabling intelligent code completion for software development tasks.

- **Base Model:** GPT-2
- **Task:** Causal Language Modeling
- **Language:** Python
- **Framework:** PyTorch
- **Library:** Hugging Face Transformers

---

## Dataset

**Dataset:** CodeXGLUE – Python Code Completion

The dataset contains Python source code snippets used to train language models for next-token code prediction.

Note: A subset of approximately 13,000 training samples from the CodeXGLUE Python dataset was used for fine-tuning.

---

## Training Configuration

| Parameter | Value |
|-----------|--------|
| Model | GPT-2 |
| Epochs | 3 |
| Learning Rate | 2e-4 |
| Batch Size | 4 |
| Gradient Accumulation | 4 |
| Weight Decay | 0.01 |
| Max Sequence Length | 512 |
| Optimizer | AdamW |
| Framework | PyTorch |

Training was performed using the Hugging Face `Trainer` API.

---

## Evaluation Results

| Metric | Value |
|---------|---------|
| Validation Loss | **1.1869** |
| Perplexity | **3.28** |

The decreasing validation loss throughout training indicates successful adaptation of GPT-2 to the Python code completion task.

---

## Training Progress

| Step | Training Loss | Validation Loss |
|------|---------------|----------------|
|100|1.5613|1.3592|
|200|1.3962|1.2877|
|300|1.3317|1.2537|
|400|1.2437|1.2308|
|500|1.2253|1.2142|
|600|1.2014|1.2000|
|Final|—|**1.1869**|

---

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/MODEL_NAME")
model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/MODEL_NAME")

prompt = "def fibonacci(n):"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Limitations

- Trained only on Python source code.
- Intended for research and educational purposes.
- May generate syntactically incorrect or incomplete code.
- Does not guarantee production-quality code suggestions.

---

## Technologies Used

- Python
- PyTorch
- Hugging Face Transformers
- Hugging Face Datasets
- CodeXGLUE Dataset

---

## Future Improvements

- Fine-tune larger transformer models.
- Train on larger subsets of CodeXGLUE.
- Evaluate using additional code generation metrics.
- Support multiple programming languages.
- Deploy as an inference API.

---

## Author

**Sai Nandu Vajhala**

GitHub: https://github.com/SaiNanduVajhala

LinkedIn: https://www.linkedin.com/in/sai-nandu-vajhala