README.md

---
library_name: transformers
tags:
- medical
- q&a
- pubmedqa
- diffusiongemma
- lora
- unsloth
license: apache-2.0
datasets:
- qiaojin/PubMedQA
language:
- en
metrics:
- accuracy
base_model:
- unsloth/diffusiongemma-26B-A4B-it
pipeline_tag: text-generation
---

# DiffusionGemma PubMedQA

This model is a LoRA fine-tuned version of **DiffusionGemma 26B-A4B IT** for biomedical question answering on **PubMedQA**.

The model was fine-tuned to answer biomedical research questions using PubMed abstract context and return one of three labels:

```text
yes / no / maybe
```

This model is intended for experimentation, benchmarking, and educational use. It is **not** intended for clinical decision-making or medical advice.

## Model Details

### Model Description

This model adapts DiffusionGemma to the PubMedQA task. Given a biomedical research question and supporting PubMed abstract context, the model predicts whether the answer is `yes`, `no`, or `maybe`.

- **Developed by:** kingabzpro
- **Shared by:** kingabzpro
- **Model type:** Diffusion language model with LoRA adapter
- **Language(s):** English
- **License:** Apache 2.0
- **Fine-tuned from:** `unsloth/diffusiongemma-26B-A4B-it`
- **Training framework:** Unsloth + Transformers
- **Task:** Biomedical question answering / text generation
- **Dataset:** `qiaojin/PubMedQA`

### Model Sources

- **Repository:** `kingabzpro/diffusiongemma_pubmedqa`
- **Base model:** `unsloth/diffusiongemma-26B-A4B-it`
- **Dataset:** `qiaojin/PubMedQA`

## Uses

### Direct Use

This model can be used to answer PubMedQA-style biomedical research questions where the input includes:

1. A biomedical research question
2. Relevant abstract/context text
3. A request to answer with `yes`, `no`, or `maybe`

Example task format:

```text
Answer the biomedical research question using only the context.

Context:
[PubMed abstract context]

Question:
[Biomedical research question]

Answer with only one word: yes, no, or maybe.
```

### Downstream Use

This model may be useful for:

- Biomedical QA experiments
- PubMedQA-style benchmark testing
- Fine-tuning tutorials
- LoRA adapter experiments with DiffusionGemma
- Educational demos for medical-domain model adaptation

### Out-of-Scope Use

This model should **not** be used for:

- Medical diagnosis
- Treatment recommendations
- Emergency medical advice
- Replacing a doctor, pharmacist, or clinical expert
- Patient-specific medical decisions
- High-stakes biomedical or healthcare deployment without further validation

The model is trained on a narrow benchmark-style task and may produce incorrect answers.

## Bias, Risks, and Limitations

This model has several important limitations:

- It was fine-tuned on PubMedQA-style examples, not general medical conversations.
- The model predicts only `yes`, `no`, or `maybe`, so it may oversimplify complex biomedical findings.
- The evaluation set used in this experiment was small: 50 examples.
- The model may be sensitive to prompt format.
- The model may answer incorrectly if the context is incomplete, misleading, or unrelated.
- The training target was short, so the loss dropped quickly and may not reflect deep medical reasoning.
- The model should not be treated as medically reliable.

### Recommendations

Users should:

- Use this model only for research and educational experiments.
- Always verify outputs against trusted biomedical sources.
- Avoid using the model for real clinical or patient-facing decisions.
- Run larger evaluations before drawing strong conclusions.
- Consider training on explanations, not only one-word labels, for a more meaningful medical QA setup.

## How to Get Started with the Model

### Install dependencies

```bash
pip install unsloth
pip install transformers datasets peft accelerate sentencepiece protobuf
```

### Load the model

If this repository contains the LoRA adapter, load the base model first and then attach the adapter:

```python
import copy
import torch
from peft import PeftModel
from unsloth import FastModel

base_model_name = "unsloth/diffusiongemma-26B-A4B-it"
adapter_name = "kingabzpro/diffusiongemma_pubmedqa"

model, tokenizer = FastModel.from_pretrained(
    model_name=base_model_name,
    dtype=torch.bfloat16,
    load_in_4bit=False,
)

model = PeftModel.from_pretrained(model, adapter_name)

processor = tokenizer
tok = processor.tokenizer if hasattr(processor, "tokenizer") else processor

dev = next(
    (p.device for p in model.parameters() if p.device.type != "meta"),
    torch.device("cuda"),
)

canvas_len = model.config.canvas_length
```

### Run inference

```python
def answer_question(prompt, steps=16):
    input_ids = processor.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    ).to(dev)

    gen_config = copy.deepcopy(model.generation_config)
    gen_config.max_denoising_steps = steps
    gen_config.max_new_tokens = canvas_len

    model.eval()

    with torch.no_grad():
        output = model.generate(
            input_ids=input_ids,
            generation_config=gen_config,
        )

    generated = output.sequences[0, input_ids.shape[1]:]
    text = tok.decode(generated.tolist(), skip_special_tokens=True)

    return text.strip().lower()


prompt = """Answer the biomedical research question using only the context.

Context:
[Paste PubMed abstract context here]

Question:
[Paste biomedical question here]

Answer with only one word: yes, no, or maybe."""

print(answer_question(prompt, steps=16))
```

## Training Details

### Training Data

The model was fine-tuned on `qiaojin/PubMedQA`.

The notebook used:

- **Training split:** `pqa_artificial`
- **Evaluation split:** `pqa_labeled`
- **Training examples used:** 3,000
- **Evaluation examples prepared:** 200
- **Evaluation examples used for reported result:** 50

Each training example was converted into a prompt-answer pair:

```text
Input:
Biomedical question + PubMed abstract context

Target:
yes / no / maybe
```

### Training Procedure

#### Preprocessing

For each PubMedQA row:

1. The abstract contexts were joined into one context block.
2. The context was truncated to 2,500 characters.
3. The question was inserted below the context.
4. The target answer was the `final_decision` field.
5. Only examples with `yes`, `no`, or `maybe` labels were used.

Prompt format:

```text
Answer the biomedical research question using only the context.

Context:
{context}

Question:
{question}

Answer with only one word: yes, no, or maybe.
```

Target format:

```text
{final_decision}
```

#### Training Hyperparameters

- **Training regime:** bf16
- **LoRA rank:** 64
- **LoRA alpha:** 128
- **Trainable parameters:** 149,630,976
- **Total parameters:** 25,973,409,840
- **Trainable percentage:** 0.5761%
- **Training examples:** 3,000
- **Training steps:** 60
- **Gradient accumulation:** 4
- **Learning rate:** 1e-4
- **Optimizer:** AdamW
- **Scheduler:** OneCycleLR
- **Weight decay:** 0.0
- **Max context characters:** 2,500
- **Canvas length:** 256
- **Dataset:** `qiaojin/PubMedQA`

#### Speeds, Sizes, Times

Training was run on a RunPod H100 notebook.

Training logs from the saved notebook:

```text
step 20/60 | loss 0.0019 | 43s
step 40/60 | loss 0.0003 | 85s
step 60/60 | loss 0.0001 | 126s
```

Approximate training time:

```text
126 seconds for 60 steps
```

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

Evaluation used the `pqa_labeled` subset of `qiaojin/PubMedQA`.

The reported run used:

- **Evaluation examples:** 50
- **Denoising steps:** 16
- **Metric:** Accuracy

#### Factors

The evaluation was not disaggregated by biomedical topic, article type, answer class, or question type. Results should be treated as a small sanity-check evaluation, not a full benchmark.

#### Metrics

Accuracy was used because PubMedQA final decisions are discrete labels:

```text
yes / no / maybe
```

A prediction was counted as correct if the cleaned model output matched the gold `final_decision`.

### Results

| Setting | Accuracy | Correct / Total |
| --- | --- | --- |
| Before fine-tuning | 0.60 | 30 / 50 |
| After fine-tuning | 0.80 | 40 / 50 |
| Improvement | +0.20 | +10 / 50 |

#### Summary

In the saved RunPod H100 notebook run, the model improved from **60% accuracy before fine-tuning** to **80% accuracy after fine-tuning** on a 50-example PubMedQA evaluation sample.

This is a **+20 percentage point improvement**.

The result shows that the model can quickly adapt to the PubMedQA answer format. However, this is a small evaluation and should not be interpreted as a clinically meaningful benchmark.

## Model Examination

No detailed interpretability or model examination was performed.

## Environmental Impact

Carbon emissions were not measured for this run.

- **Hardware Type:** NVIDIA H100 80GB HBM3
- **Hours used:** Approximately 0.04 hours for the 60-step training loop, excluding setup, model loading, and evaluation
- **Cloud Provider:** RunPod
- **Compute Region:** Not recorded
- **Carbon Emitted:** Not measured

## Technical Specifications

### Model Architecture and Objective

The base model is DiffusionGemma 26B-A4B IT, a diffusion-style language model. The fine-tuning used LoRA adapters.

The training objective followed a block-diffusion setup:

1. Encode the target answer into the model canvas.
2. Randomly corrupt answer tokens.
3. Train the model to reconstruct the clean answer.
4. Apply loss only over the target answer tokens.

### Compute Infrastructure

#### Hardware

- NVIDIA H100 80GB HBM3
- Reported GPU memory: approximately 85 GB total

#### Software

- Python
- PyTorch 2.10.0+cu128
- Transformers
- Unsloth
- Unsloth Zoo
- PEFT
- Datasets
- RunPod Jupyter Notebook

## Citation

If you use this model, please cite the original PubMedQA dataset and DiffusionGemma base model.

**PubMedQA:**

```bibtex
@inproceedings{jin2019pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William W. and Lu, Xinghua},
  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing},
  year={2019}
}
```

## Glossary

- **PubMedQA:** A biomedical question-answering dataset based on PubMed abstracts.
- **LoRA:** Low-Rank Adaptation, a parameter-efficient fine-tuning method.
- **DiffusionGemma:** A diffusion-style language model.
- **Denoising steps:** Iterative generation steps used by diffusion models.
- **Accuracy:** Percentage of predictions matching the gold label.

## More Information

This model was created as a simple fine-tuning experiment for adapting DiffusionGemma to a medical QA dataset.

The task is intentionally simple:

```text
Biomedical context + question → yes / no / maybe
```

For a stronger medical QA model, future versions should train on both:

```text
Decision: yes/no/maybe
Explanation: short evidence-based explanation
```

## Model Card Authors

- kingabzpro

## Model Card Contact

For questions, contact the model repository owner on Hugging Face.