David0dods/codeforces-gpt-neo-lora

Codeforces GPT-Neo Python Assistant

Model Description

This model is a specialized version of EleutherAI/gpt-neo-125M, fine-tuned to assist in solving competitive programming challenges. It has been trained to generate structured Python solutions based on complex problem descriptions. This project was developed as a key task for the Generative AI track within the Digital Egypt Pioneers Initiative (DEPI) scholarship.

Intended Uses & Limitations

• Intended Use: Assistance with Python coding logic and competitive programming patterns.
• Limitations: Due to its lightweight architecture (125M parameters), the model may struggle with highly complex algorithmic logic and is best used as a coding assistant rather than a primary solver.

Training and Evaluation Data

The model was fine-tuned on a specialized subset of the MatrixStudio/Codeforces-Python-Submissions dataset. The training data followed an Instruction-Problem-Solution format to ensure the model learns to follow specific programming prompts effectively.

Training Procedure

Technique: LoRA (Low-Rank Adaptation)

To ensure efficient training, the model utilized PEFT (LoRA). This allowed for effective domain adaptation while significantly reducing the number of trainable parameters.

Training Hyperparameters

• Learning Rate: 0.0002
• Total Train Batch Size: 16 (4 batch size per device with 4 accumulation steps)
• Optimizer: AdamW with Mixed Precision (Native AMP)
• Epochs: 3
• LoRA Rank (r): 8

Training Results

The model showed consistent improvement over 3 epochs, with the final Validation Loss reaching 2.0083. The steady decrease in both Training and Validation loss indicates successful convergence without overfitting.

Training Loss	Epoch	Step	Validation Loss
2.0445	1.0	563	2.0761
2.0477	2.0	1126	2.0216
1.9682	3.0	1689	2.0083

Framework Versions

• PEFT: 0.18.1
• Transformers: 5.2.0
• Pytorch: 2.10.0+cu128
• Datasets: 4.5.0