ellyfantina/llama3-medquad-lora

🧬 Llama-3.1-8B-MedQuAD-LoRA

Llama-3.1-8B-MedQuAD-LoRA is a parameter-efficient fine-tuned version of meta-llama/Llama-3.1-8B-Instruct, trained on the MedQuAD (Medical Question Answering Dataset) using QLoRA for factual and educational biomedical question answering.

---

⚙️ Technical Overview

Setting	Value
Base model	meta-llama/Llama-3.1-8B-Instruct
Fine-tuning method	QLoRA (4-bit NF4 quantization)
LoRA rank / alpha	r = 8 / α = 16
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dropout	0.1
Precision	FP16 mixed
Batch size (effective)	8 (2 × grad accum = 4)
Learning rate	2e-4
Optimizer	AdamW (β₁ = 0.9, β₂ = 0.95)
Scheduler	Linear decay, warmup = 3 %
Max sequence length	1024
Epochs	2
GPU	NVIDIA T4 (15 GB VRAM)
Training time	≈ 8 hours
Final loss (train/val)	0.76 / 0.75

---