README.md

---
base_model: meta-llama/Llama-3.2-3B
library_name: peft
tags: [lora, qlora, code-generation, python]
datasets: [sahil2801/CodeAlpaca-20k]
license: llama3.2
---

# Python code generation — LLaMA 3.2 3B + QLoRA

**HumanEval pass@1: 40.5% → 54.1%** on the uncontaminated subset — a 13.6-point gain
from a 9.2M-parameter adapter (0.285% of the model), trained in under two hours on a
single free-tier T4.

The base model already writes correct code. It just writes it in JavaScript 42% of the
time, wrapped in markdown fences. This adapter makes Python the default and the output
directly executable.

## Results

**HumanEval, first 50 problems** (greedy decoding, deterministic):

| | Base | Fine-tuned | Δ |
|---|---|---|---|
| **Clean subset (37 problems)** | **40.5%** | **54.1%** | **+13.6 pts** |
| All 50 | 46.0% | 58.0% | +12.0 pts |

The clean subset excludes 13 problems whose function names appear in the training data
(see Contamination). It is the number to cite.

**Custom eval set** — 40 hand-written Python problems with executable unit tests, all 40
reference solutions verified to pass before use:

| | Base | Fine-tuned |
|---|---|---|
| Valid Python (free-form instruction) | 57.5% | **100%** |
| Language correct | 62.5% | **100%** |
| pass@1 | 82.5% | **90%** |

Without a specified function signature, the base model emitted non-Python for 15 of 40
problems and markdown-fenced (non-executable) output for 2 more. The adapter reaches
100% executable Python with no prompt engineering.

## Usage

The prompt format is load-bearing.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True)

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B", quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, "Raghul09/llama-code-gen-lora")
tok = AutoTokenizer.from_pretrained("Raghul09/llama-code-gen-lora")

prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
```

## Training

| | |
|---|---|
| Base | meta-llama/Llama-3.2-3B |
| Quantization | 4-bit NF4 + double quant — 6.43 GB → 2.20 GB (66%) |
| LoRA | r=16, α=32, q/k/v/o across 28 layers |
| Trainable | 9,175,040 / 3,221,924,864 (0.285%) |
| Data | CodeAlpaca-20K, AST-filtered to Python — 6,418 / 802 / 803 |
| max_length | 256, set from measured distribution (p50=89, p95=211) |
| Hardware | single T4, 1.88 h, **4.17 GB peak VRAM** |
| Checkpoint | epoch 2 of 3 — val loss 0.475; epoch 3 rose to 0.485 |

Full fine-tuning of this model requires roughly 50 GB. QLoRA brought it to 4.17 GB,
a 12x reduction, which is what made it feasible on free-tier hardware.

## Rank sweep

| Rank | Trainable | Val loss | Valid Python | pass@1 |
|---|---|---|---|---|
| 8 | 4,587,520 | 0.4795 | 100% | 82.5% |
| **16** | **9,175,040** | **0.4753** | **100%** | **90.0%** |
| 32 | 18,350,080 | 0.4739 | 100% | 82.5% |

Quadrupling rank bought 1.2% lower validation loss and no consistent gain in pass@1.
The target behaviour is genuinely low-rank — consistent with the LoRA paper's central
hypothesis, tested here rather than assumed.

## Contamination

13 of the 50 HumanEval problems have their function names defined in CodeAlpaca. The
fine-tuned model scores 69.2% on those versus ~52% on clean problems, under both prompt
formats tested. Headline numbers use the clean subset only.

Function-name matching catches exact reuse but misses paraphrased problems, so 26% is a
lower bound on overlap, not an estimate.

## Limitations

- **Synthetic training data.** CodeAlpaca-20K is GPT-generated via self-instruct,
  unverified, stylistically homogeneous. Quality is bounded by the teacher model.
- **Conformance over capability.** With a function signature specified in the prompt,
  the base model already reaches 97.5% valid Python and 82.5% pass@1. This adapter's
  main contribution is removing the need for that prompt engineering.
- **Python only.** Training data was AST-filtered to Python.
- **Short outputs.** Median training example was 89 tokens; long generations degrade.
- **Known failure modes:** repetition loops causing mid-generation truncation; calling
  helper functions it never defines.

## Methodology notes

Three measurement errors found and corrected during evaluation:

1. **Naming confound.** Initial pass@1 read 35% base / 50% fine-tuned. 17 of 20 failures
   were `NameError` — correct code under a different function name than the test called.
   Specifying signatures corrected the baseline by 47 points, to 82.5%.

2. **Indentation destruction.** Raw-format HumanEval returned 0% for both models. The
   fence-stripper called `.strip()`, removing leading indentation from function bodies.
   All 50 failures were `IndentationError`.

3. **Training-data contamination.** A keyword-based Python filter kept 58.7% of
   CodeAlpaca; AST-parsing a 300-example sample showed 38% weren't valid Python — mostly
   Java and JavaScript matching on shared keywords like `for` and `class`. Replaced with
   `ast.parse` plus a syntax-tree check: 0% contamination on re-check.

Sequence packing was disabled after batch inspection revealed it silently disabled
completion-only loss masking. Packing requires Flash Attention for block-diagonal
masking, which requires Ampere; the T4 is Turing.