README.md

---
library_name: peft
license: llama3.2
base_model: meta-llama/Llama-3.2-3B-Instruct
tags:
- generated_from_trainer
datasets:
- demoversion/cf-cpp-to-python-code-generation
model-index:
- name: outputs/cf-llm-finetune-llama-3.2-3b-lora
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
<details><summary>See axolotl config</summary>

axolotl version: `0.10.0`
```yaml
base_model: meta-llama/Llama-3.2-3B-Instruct

load_in_8bit: true
load_in_4bit: false

datasets:
  - path: ./data/train_openai_response_transformed.jsonl
    type: chat_template

    field_messages: messages
    message_property_mappings:
      role: role
      content: content

val_file: ./data/val_openai_response_transformed.jsonl
val_set_size: 0.0
output_dir: ./outputs/cf-llm-finetune-llama-3.2-3b-lora

adapter: lora
lora_model_dir:

sequence_len: 4096
sample_packing: false
eval_sample_packing: true
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4

optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

bf16: auto
tf32: false

gradient_checkpointing: true
resume_from_checkpoint:
logging_steps: 1
flash_attention: false

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.0
special_tokens:
  pad_token: "<|end_of_text|>"

```

</details><br>


# Llama-3.2-3B-Instruct-PEFT-code-generation

This model is a fine tuned [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) on a synthetic dataset of C++ → Python code translations from Codeforces.

📦 GitHub repo: [DemoVersion/cf-llm-finetune](https://github.com/DemoVersion/cf-llm-finetune)

📑 Dataset Creation [DATASET.md](https://github.com/DemoVersion/cf-llm-finetune/blob/main/DATASET.md)

📑 Training [TRAIN.md](https://github.com/DemoVersion/cf-llm-finetune/blob/main/TRAIN.md)  

📚 Dataset on Hugging Face: [demoversion/cf-cpp-to-python-code-generation](https://huggingface.co/datasets/demoversion/cf-cpp-to-python-code-generation)

For dataset generation, training, and inference check the [Github repo](https://github.com/DemoVersion/cf-llm-finetune).

**📚 Main medium article**: [Toward fine-tuning Llama 3.2 using PEFT for Code Generation](https://medium.com/@haddadhesam/towards-fine-tuning-llama-3-2-using-peft-for-code-generation-63e3991c26db)

**📚 Medium article for inference with GGUF format**: [How to inference with GGUF format](https://haddadhesam.medium.com/one-file-to-rule-them-all-gguf-for-local-llm-testing-and-deployment-208b85934434)

## Model description

A lightweight LLaMA 3.2 model fine-tuned for competitive programming code translation, from ICPC-style C++ to Python using LoRA adapters.

## Intended uses & limitations

**Use for:**
-   Translating competitive programming C++ solutions to Python
-   Code understanding in educational or automation tools
    

**Limitations:**

-   Not general-purpose code translation
-   Python outputs are synthetically generated using GPT-4.1
-   Focused only on ICPC-style problems
    

## Training and evaluation data

Training and Evaluation data:  
🧾 [demoversion/cf-cpp-to-python-code-generation](https://huggingface.co/datasets/demoversion/cf-cpp-to-python-code-generation)

Built from:
-   [open-r1/codeforces-submissions](https://huggingface.co/datasets/open-r1/codeforces-submissions)
-   [open-r1/codeforces](https://huggingface.co/datasets/open-r1/codeforces)

C++ submissions were filtered and paired with GPT-4.1-generated Python translations. Dataset split: 1,400 train / 300 val / 300 test. To underestand how the dataset was created check [DATASET.md](https://github.com/DemoVersion/cf-llm-finetune/blob/main/DATASET.md)

## Training procedure
-   Adapter: LoRA (`r=32`, `alpha=16`, `dropout=0.05`)
-   Optimizer: `adamw_bnb_8bit`
-   LR: `2e-4`, scheduler: `cosine`
-   Batch size: 2 × 4 (grad accumulation) = total 8
-   Training steps: 688  
    Full config: [TRAIN.md](https://github.com/DemoVersion/cf-llm-finetune/blob/main/TRAIN.md)
    

## Framework versions
-   PEFT 0.15.2
-   Transformers 4.52.3
-   PyTorch 2.6.0+cu124
-   Datasets 3.6.0
-   Tokenizers 0.21.2