gpt2-medqa-lora

A LoRA adapter for GPT-2 (124M), fine-tuned for one epoch on MedQuAD medical Q&A. It is the baseline arm of a controlled comparison of LoRA against QLoRA — the other arm is Babblu2821/tinyllama-medqa-qlora.

Code, method and full results: https://github.com/fayazhussain2821/llm-finetuning-medqa

⚠️ Do not use this for medical information

This is a methodology demonstration, not a medical tool. Its factual reliability has been measured, and it is poor: on a blinded review of 20 held-out questions, this adapter contradicted the reference answer or invented an entity in 60% of them, scoring 1.80 out of 5 for factual soundness.

It produces fluent, confident, well-formed text that is usually wrong. Observed failures include attributing Marfan syndrome to "an infection" and inventing non-existent genes and citations. Fluency is exactly what makes this dangerous.

Do not use it for diagnosis, treatment, triage, patient-facing text, or to answer any real health question.

What it is for

Reproducing and studying a parameter-efficient fine-tuning comparison. The adapter is useful as an object of measurement — it is the smaller, cheaper arm that the project's controls are measured against. It is not useful as a question-answering model.

Training

Base model	gpt2 (124M)
Method	LoRA (r=16, α=32, dropout=0.05)
Target modules	c_attn, c_proj
Data	MedQuAD, 16,407 pairs, 90/10 split, seed 42 → 14,766 train
Epochs	1
Learning rate	2e-4, cosine schedule, warmup ratio 0.03
Effective batch	16 (8 × 2 accumulation) — matched to the QLoRA arm
Max length	1024 tokens
Prompt format	### Instruction:\n{question}\n\n### Response:\n
Hardware	Colab T4

Provenance. These weights were trained on 2026-08-03 with the project's original notebook pipeline, before the code was restructured into a package. The current repository trains both arms under transformers.Trainer (the notebooks used TRL's SFTTrainer for the TinyLlama arm), so re-running the current code will not reproduce these exact weights. Every published number below was measured on these files.

Evaluation

Scored on 1,641 held-out rows, identical rows for every arm, answer span only — the prompt template differs between arms and scoring it would let boilerplate move the metric.

Bits per byte is the headline metric, not perplexity. Perplexity is per token, and GPT-2's tokenizer differs from TinyLlama's, so the two perplexities are not on one scale. Bits per byte normalises by UTF-8 bytes of the same reference text.

run	bits/byte ↓	perplexity
gpt2 (untrained control)	0.8049	11.51
gpt2-medqa-lora (this model)	0.5970	6.12
tinyllama (untrained control)	0.6120	5.39
tinyllama-medqa-qlora	0.3954	2.97

Fine-tuning cut bits per byte by 25.8% against its own base model.

Generated-answer quality

Greedy decoding, ≤200 new tokens, 200 held-out questions:

	ROUGE-L F1 ↑	token F1 ↑	repeated 4-grams ↓
gpt2 (control)	0.0797	0.1666	0.0000
this model	0.0971	0.2060	0.0005

No degeneration — the model does not loop. Its problem is that it is wrong.

Factual soundness (blinded, 1–5)

	mean ↑	contradicts reference ↓
gpt2 (control)	1.60	70%
this model	1.80	60%

Fine-tuning produced no detectable improvement in factual accuracy (paired 95% CI −0.40 to +0.80, spanning zero) — while the automatic metrics above reported a 25.8% gain. LoRA taught this model MedQuAD's register, and register is what those metrics score.

These ratings are a human pass, rated blind to which model produced each answer, by the repository's author — one non-expert rater, not a clinician and not adjudicated by a second. An earlier LLM-judge pass over the same sheet rated this adapter far harsher (1.20 mean, 95% contradiction) but reached every identical verdict; both are published in the source repository.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("gpt2")
model = PeftModel.from_pretrained(base, "Babblu2821/gpt2-medqa-lora")
tokenizer = AutoTokenizer.from_pretrained("Babblu2821/gpt2-medqa-lora")

prompt = "### Instruction:\nWhat is anemia?\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=160, repetition_penalty=1.15)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The prompt template matters: this adapter was trained on ### Instruction: / ### Response: and will behave worse without it.

Limitations

• Not factually reliable. See the measured numbers above.
• One epoch, one seed, one run — no variance estimate across training runs.
• MedQuAD is NIH-sourced, US-centric, and frozen at collection time.
• ~5% of examples exceed GPT-2's 1024-token context and were truncated.
• Evaluation compares against a single reference answer; a correct answer phrased differently scores as a miss.

License

MIT for the adapter weights. The base model and dataset carry their own licenses.