docs: real model card — measured results, provenance, and safety limits

README.md

@@ -1,62 +1,163 @@
 ---
 base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
 library_name: peft
-model_name: tinyllama-medqa-qlora
+pipeline_tag: text-generation
+license: apache-2.0
+language:
+- en
+datasets:
+- keivalya/MedQuad-MedicalQnADataset
 tags:
 - base_model:adapter:TinyLlama/TinyLlama-1.1B-Chat-v1.0
 - lora
-- sft
+- qlora
+- peft
 - transformers
-- trl
-licence: license
-pipeline_tag: text-generation
+- medical
+- research-artifact
 ---
 
-# Model Card for tinyllama-medqa-qlora
-
-This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0).
-It has been trained using [TRL](https://github.com/huggingface/trl).
-
-## Quick start
+# tinyllama-medqa-qlora
+
+A QLoRA adapter (4-bit NF4, double-quantised) for **TinyLlama-1.1B-Chat**, fine-tuned
+for one epoch on [MedQuAD](https://huggingface.co/datasets/keivalya/MedQuad-MedicalQnADataset)
+medical Q&A. It is the **treatment arm** of a controlled comparison against LoRA on a
+smaller model — the other arm is
+[`Babblu2821/gpt2-medqa-lora`](https://huggingface.co/Babblu2821/gpt2-medqa-lora).
+
+Code, method and full results: **https://github.com/fayazhussain2821/llm-finetuning-medqa**
+
+> ## ⚠️ Do not use this for medical information
+>
+> This is a **methodology demonstration**, not a medical tool. Its factual reliability
+> has been measured, and while it is the best arm in this project, it is still poor: on
+> a blinded review of 20 held-out questions it **contradicted the reference answer or
+> invented an entity in 40% of them**, scoring **2.70 out of 5** for factual soundness.
+>
+> It produces fluent, confident, well-formed text that is frequently wrong. Fluency is
+> exactly what makes this dangerous.
+>
+> Do not use it for diagnosis, treatment, triage, patient-facing text, or to answer any
+> real health question.
+
+## What it is for
+
+Reproducing and studying a parameter-efficient fine-tuning comparison. Note that the
+project's own conclusion is a caution against over-reading this adapter: most of its
+advantage over the LoRA arm comes from the base model being ~9× larger and already
+instruction-tuned, not from QLoRA. See "Evaluation".
+
+## Training
+
+| | |
+|---|---|
+| Base model | `TinyLlama/TinyLlama-1.1B-Chat-v1.0` (1.1B) |
+| Method | QLoRA — 4-bit NF4, double quantisation, fp16 compute |
+| LoRA | r=16, α=32, dropout=0.05 |
+| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
+| Data | MedQuAD, 16,407 pairs, 90/10 split, seed 42 → 14,766 train |
+| Epochs | 1 |
+| Learning rate | 2e-4, cosine schedule, 30 warmup steps |
+| Effective batch | 16 (4 × 4 accumulation) — matched to the LoRA arm |
+| Optimiser | `paged_adamw_8bit` |
+| Mixed precision | Off — QLoRA trains fp32 adapters with no grad scaler |
+| Max length | 1024 tokens |
+| Prompt format | TinyLlama's native chat template |
+| Hardware | Colab T4 |
+
+**Provenance.** These weights were trained on 2026-08-03 with the project's original
+notebook pipeline, using TRL's `SFTTrainer`. The current repository trains both arms
+under `transformers.Trainer` — a deliberate change, so the two arms of the comparison
+no longer run different training machinery. **Re-running the current code will not
+reproduce these exact weights.** Every published number below was measured on *these*
+files.
+
+## Evaluation
+
+Scored on 1,641 held-out rows, identical rows for every arm, **answer span only** —
+the prompt template differs between arms and scoring it would let boilerplate move the
+metric.
+
+**Bits per byte** is the headline metric, not perplexity. Perplexity is per *token*,
+and TinyLlama's SentencePiece tokenizer differs from GPT-2's byte-level BPE, so the two
+perplexities are not on one scale.
+
+| run | bits/byte ↓ | perplexity |
+|---|---|---|
+| `gpt2` (untrained control) | 0.8049 | 11.51 |
+| `gpt2-medqa-lora` | 0.5970 | 6.12 |
+| `TinyLlama-1.1B-Chat` (untrained control) | 0.6120 | 5.39 |
+| **`tinyllama-medqa-qlora` (this model)** | **0.3954** | **2.97** |
+
+Fine-tuning cut bits per byte by **35.4%** against its own base model.
+
+**Read the control row carefully.** Untouched TinyLlama (0.6120) is already within a
+few percent of *fully fine-tuned* GPT-2 (0.5970), having seen no MedQuAD at all. Most
+of the cross-model gap is model scale and prior instruction tuning, **not** QLoRA. An
+earlier version of this project reported a "53% improvement" by comparing the two
+fine-tuned arms' perplexities directly; that number was wrong twice over — different
+tokenizers, and no control.
+
+### Generated-answer quality
+
+Greedy decoding, ≤200 new tokens, 200 held-out questions:
+
+| | ROUGE-L F1 ↑ | token F1 ↑ | repeated 4-grams ↓ |
+|---|---|---|---|
+| base (control) | 0.1548 | 0.2718 | 0.0101 |
+| **this model** | **0.2337** | **0.3435** | **0.0146** |
+
+No degeneration — the model does not loop.
+
+### Factual soundness (blinded, 1–5)
+
+| | mean ↑ | contradicts reference ↓ |
+|---|---|---|
+| base (control) | 2.30 | 50% |
+| **this model** | **2.70** | **40%** |
+
+Fine-tuning produced **no detectable improvement in factual accuracy** (paired 95% CI
+−0.10 to +0.90, spanning zero), despite the automatic metrics reporting large gains.
+What *is* detectable is that the untouched base model already beats the fully
+fine-tuned GPT-2 arm by +1.10 (95% CI +0.55 to +1.65).
+
+These ratings were produced by an LLM judge, not a clinician, and are labelled as such
+in the source repository. They are a strong prior, not a clinical assessment.
+
+## Usage
 
 ```python
-from transformers import pipeline
-
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="None", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
+base = AutoModelForCausalLM.from_pretrained(base_id)
+model = PeftModel.from_pretrained(base, "Babblu2821/tinyllama-medqa-qlora")
+tokenizer = AutoTokenizer.from_pretrained("Babblu2821/tinyllama-medqa-qlora")
+
+prompt = tokenizer.apply_chat_template(
+    [{"role": "user", "content": "What is anemia?"}],
+    tokenize=False,
+    add_generation_prompt=True,
+)
+inputs = tokenizer(prompt, return_tensors="pt")
+out = model.generate(**inputs, max_new_tokens=160, repetition_penalty=1.15)
+print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
 
-## Training procedure
-
- 
-
-
-
-This model was trained with SFT.
-
-### Framework versions
-
-- PEFT 0.19.1
-- TRL: 1.9.2
-- Transformers: 5.13.1
-- Pytorch: 2.11.0+cu128
-- Datasets: 5.0.1
-- Tokenizers: 0.22.2
+4-bit loading requires CUDA (`bitsandbytes`). On CPU or Apple Silicon, load in fp16 or
+fp32 as above — the published evaluation was run that way, in fp16 on MPS, so the
+figures are not bit-identical to a 4-bit run.
 
-## Citations
+## Limitations
 
+- **Not factually reliable.** See the measured numbers above.
+- One epoch, one seed, one run — no variance estimate across training runs.
+- The comparison against the GPT-2 arm confounds model size, prior instruction tuning
+  and adaptation method. Only the within-model comparison is attributable.
+- MedQuAD is NIH-sourced, US-centric, and frozen at collection time.
+- Evaluation compares against a single reference answer.
 
+## License
 
-Cite TRL as:
-    
-```bibtex
-@software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
-}
-```
+Apache 2.0, following the base model. The dataset carries its own license.