llama-3-8b-lora-code-gen

Use case

Instruction-following code generation assistant. Given a natural-language instruction (optionally with an input), the model produces the corresponding code. Fine-tuned as part of a structured AI/ML engineering roadmap (Day 23-25) to test whether LoRA fine-tuning improves code-generation quality over the base Llama-3-8B model.

Base model

unsloth/llama-3-8b-bnb-4bit (4-bit quantized Llama-3-8B)

Training data

FathyElghoneimy/alpaca-code-generation-curated — a curated, Alpaca-formatted code-generation dataset (80/10/10 train/validation/test split, 640 training examples).

Training procedure

• Method: LoRA (r=16, alpha=32, dropout=0) via Unsloth + TRL's SFTTrainer
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• 4 epochs, batch size 2 (grad accumulation 4 -> effective batch 8), lr 2e-4, linear schedule, 4-bit base weights, fp16

Evaluation

Evaluated on 50 held-out test prompts against the base model. Metrics: ROUGE-L, BERTScore-F1, and an LLM-as-judge rubric (Claude, correctness/completeness/quality, 1-5).

Metric	Base model	Fine-tuned model	Delta
ROUGE-L	0.2191	0.3897	+0.1706
BERTScore-F1	0.8436	0.9064	+0.0629
Judge: correctness	3.5714	4.2857	+0.7143
Judge: completeness	3.1429	3.8571	+0.7143
Judge: quality	3.0000	4.0000	+1.0000

Fine-tuned model was judged better overall on 8.0% of test prompts.

Limitations

• Trained on a relatively small curated dataset (640 examples); may not generalize to code domains outside the training distribution.
• Evaluated with an LLM-as-judge, which itself is an approximation of true code quality — not a substitute for unit tests or human code review.
• 4-bit quantized base model trades some precision for lower VRAM use; full-precision behavior may differ.
• No adversarial or safety-specific evaluation was performed; do not use for unreviewed production code generation.

How to use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="FathyElghoneimy/llama-3-8b-lora-code-gen",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)