README.md

---
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- en
datasets:
- keivalya/MedQuad-MedicalQnADataset
tags:
- base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
- lora
- qlora
- peft
- transformers
- medical
- research-artifact
---

# tinyllama-medqa-qlora

A QLoRA adapter (4-bit NF4, double-quantised) for **TinyLlama-1.1B-Chat**, fine-tuned
for one epoch on [MedQuAD](https://huggingface.co/datasets/keivalya/MedQuad-MedicalQnADataset)
medical Q&A. It is the **treatment arm** of a controlled comparison against LoRA on a
smaller model — the other arm is
[`Babblu2821/gpt2-medqa-lora`](https://huggingface.co/Babblu2821/gpt2-medqa-lora).

Code, method and full results: **https://github.com/fayazhussain2821/llm-finetuning-medqa**

> ## ⚠️ Do not use this for medical information
>
> This is a **methodology demonstration**, not a medical tool. Its factual reliability
> has been measured, and while it is the best arm in this project, it is still poor: on
> a blinded review of 20 held-out questions it **contradicted the reference answer or
> invented an entity in 35% of them**, scoring **3.25 out of 5** for factual soundness.
>
> It produces fluent, confident, well-formed text that is frequently wrong. Fluency is
> exactly what makes this dangerous.
>
> Do not use it for diagnosis, treatment, triage, patient-facing text, or to answer any
> real health question.

## What it is for

Reproducing and studying a parameter-efficient fine-tuning comparison. Note that the
project's own conclusion is a caution against over-reading this adapter: most of its
advantage over the LoRA arm comes from the base model being ~9× larger and already
instruction-tuned, not from QLoRA. See "Evaluation".

## Training

| | |
|---|---|
| Base model | `TinyLlama/TinyLlama-1.1B-Chat-v1.0` (1.1B) |
| Method | QLoRA — 4-bit NF4, double quantisation, fp16 compute |
| LoRA | r=16, α=32, dropout=0.05 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Data | MedQuAD, 16,407 pairs, 90/10 split, seed 42 → 14,766 train |
| Epochs | 1 |
| Learning rate | 2e-4, cosine schedule, 30 warmup steps |
| Effective batch | 16 (4 × 4 accumulation) — matched to the LoRA arm |
| Optimiser | `paged_adamw_8bit` |
| Mixed precision | Off — QLoRA trains fp32 adapters with no grad scaler |
| Max length | 1024 tokens |
| Prompt format | TinyLlama's native chat template |
| Hardware | Colab T4 |

**Provenance.** These weights were trained on 2026-08-03 with the project's original
notebook pipeline, using TRL's `SFTTrainer`. The current repository trains both arms
under `transformers.Trainer` — a deliberate change, so the two arms of the comparison
no longer run different training machinery. **Re-running the current code will not
reproduce these exact weights.** Every published number below was measured on *these*
files.

## Evaluation

Scored on 1,641 held-out rows, identical rows for every arm, **answer span only** —
the prompt template differs between arms and scoring it would let boilerplate move the
metric.

**Bits per byte** is the headline metric, not perplexity. Perplexity is per *token*,
and TinyLlama's SentencePiece tokenizer differs from GPT-2's byte-level BPE, so the two
perplexities are not on one scale.

| run | bits/byte ↓ | perplexity |
|---|---|---|
| `gpt2` (untrained control) | 0.8049 | 11.51 |
| `gpt2-medqa-lora` | 0.5970 | 6.12 |
| `TinyLlama-1.1B-Chat` (untrained control) | 0.6120 | 5.39 |
| **`tinyllama-medqa-qlora` (this model)** | **0.3954** | **2.97** |

Fine-tuning cut bits per byte by **35.4%** against its own base model.

**Read the control row carefully.** Untouched TinyLlama (0.6120) is already within a
few percent of *fully fine-tuned* GPT-2 (0.5970), having seen no MedQuAD at all. Most
of the cross-model gap is model scale and prior instruction tuning, **not** QLoRA. An
earlier version of this project reported a "53% improvement" by comparing the two
fine-tuned arms' perplexities directly; that number was wrong twice over — different
tokenizers, and no control.

### Generated-answer quality

Greedy decoding, ≤200 new tokens, 200 held-out questions:

| | ROUGE-L F1 ↑ | token F1 ↑ | repeated 4-grams ↓ |
|---|---|---|---|
| base (control) | 0.1548 | 0.2718 | 0.0101 |
| **this model** | **0.2337** | **0.3435** | **0.0146** |

No degeneration — the model does not loop.

### Factual soundness (blinded, 1–5)

| | mean ↑ | contradicts reference ↓ |
|---|---|---|
| base (control) | 2.90 | 50% |
| **this model** | **3.25** | **35%** |

Fine-tuning produced **no detectable improvement in factual accuracy** (paired 95% CI
−0.20 to +0.90, spanning zero), despite the automatic metrics reporting large gains.
What *is* detectable is that the untouched base model already beats the fully
fine-tuned GPT-2 arm by +1.10 (95% CI +0.35 to +1.85).

These ratings are a human pass, rated blind to which model produced each answer, by
the repository's author — one non-expert rater, not a clinician and not adjudicated by
a second. An earlier LLM-judge pass over the same sheet scored every arm lower (2.70
mean, 40% contradiction for this model) but reached every identical verdict; both are
published in the source repository.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
base = AutoModelForCausalLM.from_pretrained(base_id)
model = PeftModel.from_pretrained(base, "Babblu2821/tinyllama-medqa-qlora")
tokenizer = AutoTokenizer.from_pretrained("Babblu2821/tinyllama-medqa-qlora")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is anemia?"}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=160, repetition_penalty=1.15)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

4-bit loading requires CUDA (`bitsandbytes`). On CPU or Apple Silicon, load in fp16 or
fp32 as above — the published evaluation was run that way, in fp16 on MPS, so the
figures are not bit-identical to a 4-bit run.

## Limitations

- **Not factually reliable.** See the measured numbers above.
- One epoch, one seed, one run — no variance estimate across training runs.
- The comparison against the GPT-2 arm confounds model size, prior instruction tuning
  and adaptation method. Only the within-model comparison is attributable.
- MedQuAD is NIH-sourced, US-centric, and frozen at collection time.
- Evaluation compares against a single reference answer.

## License

Apache 2.0, following the base model. The dataset carries its own license.