README.md

---
base_model: gpt2
library_name: peft
pipeline_tag: text-generation
license: mit
language:
- en
datasets:
- keivalya/MedQuad-MedicalQnADataset
tags:
- base_model:adapter:gpt2
- lora
- peft
- transformers
- medical
- research-artifact
---

# gpt2-medqa-lora

A LoRA adapter for **GPT-2 (124M)**, fine-tuned for one epoch on
[MedQuAD](https://huggingface.co/datasets/keivalya/MedQuad-MedicalQnADataset) medical
Q&A. It is the **baseline arm** of a controlled comparison of LoRA against QLoRA —
the other arm is [`Babblu2821/tinyllama-medqa-qlora`](https://huggingface.co/Babblu2821/tinyllama-medqa-qlora).

Code, method and full results: **https://github.com/fayazhussain2821/llm-finetuning-medqa**

> ## ⚠️ Do not use this for medical information
>
> This is a **methodology demonstration**, not a medical tool. Its factual reliability
> has been measured, and it is poor: on a blinded review of 20 held-out questions, this
> adapter **contradicted the reference answer or invented an entity in 60% of them**,
> scoring **1.80 out of 5** for factual soundness.
>
> It produces fluent, confident, well-formed text that is usually wrong. Observed
> failures include attributing Marfan syndrome to "an infection" and inventing
> non-existent genes and citations. Fluency is exactly what makes this dangerous.
>
> Do not use it for diagnosis, treatment, triage, patient-facing text, or to answer
> any real health question.

## What it is for

Reproducing and studying a parameter-efficient fine-tuning comparison. The adapter is
useful as an *object of measurement* — it is the smaller, cheaper arm that the project's
controls are measured against. It is not useful as a question-answering model.

## Training

| | |
|---|---|
| Base model | `gpt2` (124M) |
| Method | LoRA (r=16, α=32, dropout=0.05) |
| Target modules | `c_attn`, `c_proj` |
| Data | MedQuAD, 16,407 pairs, 90/10 split, seed 42 → 14,766 train |
| Epochs | 1 |
| Learning rate | 2e-4, cosine schedule, warmup ratio 0.03 |
| Effective batch | 16 (8 × 2 accumulation) — matched to the QLoRA arm |
| Max length | 1024 tokens |
| Prompt format | `### Instruction:\n{question}\n\n### Response:\n` |
| Hardware | Colab T4 |

**Provenance.** These weights were trained on 2026-08-03 with the project's original
notebook pipeline, *before* the code was restructured into a package. The current
repository trains both arms under `transformers.Trainer` (the notebooks used TRL's
`SFTTrainer` for the TinyLlama arm), so **re-running the current code will not
reproduce these exact weights.** Every published number below was measured on *these*
files.

## Evaluation

Scored on 1,641 held-out rows, identical rows for every arm, **answer span only** —
the prompt template differs between arms and scoring it would let boilerplate move the
metric.

**Bits per byte** is the headline metric, not perplexity. Perplexity is per *token*,
and GPT-2's tokenizer differs from TinyLlama's, so the two perplexities are not on one
scale. Bits per byte normalises by UTF-8 bytes of the same reference text.

| run | bits/byte ↓ | perplexity |
|---|---|---|
| `gpt2` (untrained control) | 0.8049 | 11.51 |
| **`gpt2-medqa-lora` (this model)** | **0.5970** | **6.12** |
| `tinyllama` (untrained control) | 0.6120 | 5.39 |
| `tinyllama-medqa-qlora` | 0.3954 | 2.97 |

Fine-tuning cut bits per byte by **25.8%** against its own base model.

### Generated-answer quality

Greedy decoding, ≤200 new tokens, 200 held-out questions:

| | ROUGE-L F1 ↑ | token F1 ↑ | repeated 4-grams ↓ |
|---|---|---|---|
| `gpt2` (control) | 0.0797 | 0.1666 | 0.0000 |
| **this model** | **0.0971** | **0.2060** | **0.0005** |

No degeneration — the model does not loop. Its problem is that it is wrong.

### Factual soundness (blinded, 1–5)

| | mean ↑ | contradicts reference ↓ |
|---|---|---|
| `gpt2` (control) | 1.60 | 70% |
| **this model** | **1.80** | **60%** |

Fine-tuning produced **no detectable improvement in factual accuracy** (paired 95% CI
−0.40 to +0.80, spanning zero) — while the automatic metrics above reported a 25.8%
gain. LoRA taught this model MedQuAD's *register*, and register is what those metrics
score.

These ratings are a human pass, rated blind to which model produced each answer, by
the repository's author — one non-expert rater, not a clinician and not adjudicated by
a second. An earlier LLM-judge pass over the same sheet rated this adapter far harsher
(1.20 mean, 95% contradiction) but reached every identical verdict; both are published
in the source repository.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("gpt2")
model = PeftModel.from_pretrained(base, "Babblu2821/gpt2-medqa-lora")
tokenizer = AutoTokenizer.from_pretrained("Babblu2821/gpt2-medqa-lora")

prompt = "### Instruction:\nWhat is anemia?\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=160, repetition_penalty=1.15)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

The prompt template matters: this adapter was trained on `### Instruction:` /
`### Response:` and will behave worse without it.

## Limitations

- **Not factually reliable.** See the measured numbers above.
- One epoch, one seed, one run — no variance estimate across training runs.
- MedQuAD is NIH-sourced, US-centric, and frozen at collection time.
- ~5% of examples exceed GPT-2's 1024-token context and were truncated.
- Evaluation compares against a single reference answer; a correct answer phrased
  differently scores as a miss.

## License

MIT for the adapter weights. The base model and dataset carry their own licenses.