--- library_name: transformers tags: - medical - q&a - pubmedqa - diffusiongemma - lora - unsloth license: apache-2.0 datasets: - qiaojin/PubMedQA language: - en metrics: - accuracy base_model: - unsloth/diffusiongemma-26B-A4B-it pipeline_tag: text-generation --- # DiffusionGemma PubMedQA This model is a LoRA fine-tuned version of **DiffusionGemma 26B-A4B IT** for biomedical question answering on **PubMedQA**. The model was fine-tuned to answer biomedical research questions using PubMed abstract context and return one of three labels: ```text yes / no / maybe ``` This model is intended for experimentation, benchmarking, and educational use. It is **not** intended for clinical decision-making or medical advice. ## Model Details ### Model Description This model adapts DiffusionGemma to the PubMedQA task. Given a biomedical research question and supporting PubMed abstract context, the model predicts whether the answer is `yes`, `no`, or `maybe`. - **Developed by:** kingabzpro - **Shared by:** kingabzpro - **Model type:** Diffusion language model with LoRA adapter - **Language(s):** English - **License:** Apache 2.0 - **Fine-tuned from:** `unsloth/diffusiongemma-26B-A4B-it` - **Training framework:** Unsloth + Transformers - **Task:** Biomedical question answering / text generation - **Dataset:** `qiaojin/PubMedQA` ### Model Sources - **Repository:** `kingabzpro/diffusiongemma_pubmedqa` - **Base model:** `unsloth/diffusiongemma-26B-A4B-it` - **Dataset:** `qiaojin/PubMedQA` ## Uses ### Direct Use This model can be used to answer PubMedQA-style biomedical research questions where the input includes: 1. A biomedical research question 2. Relevant abstract/context text 3. A request to answer with `yes`, `no`, or `maybe` Example task format: ```text Answer the biomedical research question using only the context. Context: [PubMed abstract context] Question: [Biomedical research question] Answer with only one word: yes, no, or maybe. ``` ### Downstream Use This model may be useful for: - Biomedical QA experiments - PubMedQA-style benchmark testing - Fine-tuning tutorials - LoRA adapter experiments with DiffusionGemma - Educational demos for medical-domain model adaptation ### Out-of-Scope Use This model should **not** be used for: - Medical diagnosis - Treatment recommendations - Emergency medical advice - Replacing a doctor, pharmacist, or clinical expert - Patient-specific medical decisions - High-stakes biomedical or healthcare deployment without further validation The model is trained on a narrow benchmark-style task and may produce incorrect answers. ## Bias, Risks, and Limitations This model has several important limitations: - It was fine-tuned on PubMedQA-style examples, not general medical conversations. - The model predicts only `yes`, `no`, or `maybe`, so it may oversimplify complex biomedical findings. - The evaluation set used in this experiment was small: 50 examples. - The model may be sensitive to prompt format. - The model may answer incorrectly if the context is incomplete, misleading, or unrelated. - The training target was short, so the loss dropped quickly and may not reflect deep medical reasoning. - The model should not be treated as medically reliable. ### Recommendations Users should: - Use this model only for research and educational experiments. - Always verify outputs against trusted biomedical sources. - Avoid using the model for real clinical or patient-facing decisions. - Run larger evaluations before drawing strong conclusions. - Consider training on explanations, not only one-word labels, for a more meaningful medical QA setup. ## How to Get Started with the Model ### Install dependencies ```bash pip install unsloth pip install transformers datasets peft accelerate sentencepiece protobuf ``` ### Load the model If this repository contains the LoRA adapter, load the base model first and then attach the adapter: ```python import copy import torch from peft import PeftModel from unsloth import FastModel base_model_name = "unsloth/diffusiongemma-26B-A4B-it" adapter_name = "kingabzpro/diffusiongemma_pubmedqa" model, tokenizer = FastModel.from_pretrained( model_name=base_model_name, dtype=torch.bfloat16, load_in_4bit=False, ) model = PeftModel.from_pretrained(model, adapter_name) processor = tokenizer tok = processor.tokenizer if hasattr(processor, "tokenizer") else processor dev = next( (p.device for p in model.parameters() if p.device.type != "meta"), torch.device("cuda"), ) canvas_len = model.config.canvas_length ``` ### Run inference ```python def answer_question(prompt, steps=16): input_ids = processor.apply_chat_template( [{"role": "user", "content": prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt", ).to(dev) gen_config = copy.deepcopy(model.generation_config) gen_config.max_denoising_steps = steps gen_config.max_new_tokens = canvas_len model.eval() with torch.no_grad(): output = model.generate( input_ids=input_ids, generation_config=gen_config, ) generated = output.sequences[0, input_ids.shape[1]:] text = tok.decode(generated.tolist(), skip_special_tokens=True) return text.strip().lower() prompt = """Answer the biomedical research question using only the context. Context: [Paste PubMed abstract context here] Question: [Paste biomedical question here] Answer with only one word: yes, no, or maybe.""" print(answer_question(prompt, steps=16)) ``` ## Training Details ### Training Data The model was fine-tuned on `qiaojin/PubMedQA`. The notebook used: - **Training split:** `pqa_artificial` - **Evaluation split:** `pqa_labeled` - **Training examples used:** 3,000 - **Evaluation examples prepared:** 200 - **Evaluation examples used for reported result:** 50 Each training example was converted into a prompt-answer pair: ```text Input: Biomedical question + PubMed abstract context Target: yes / no / maybe ``` ### Training Procedure #### Preprocessing For each PubMedQA row: 1. The abstract contexts were joined into one context block. 2. The context was truncated to 2,500 characters. 3. The question was inserted below the context. 4. The target answer was the `final_decision` field. 5. Only examples with `yes`, `no`, or `maybe` labels were used. Prompt format: ```text Answer the biomedical research question using only the context. Context: {context} Question: {question} Answer with only one word: yes, no, or maybe. ``` Target format: ```text {final_decision} ``` #### Training Hyperparameters - **Training regime:** bf16 - **LoRA rank:** 64 - **LoRA alpha:** 128 - **Trainable parameters:** 149,630,976 - **Total parameters:** 25,973,409,840 - **Trainable percentage:** 0.5761% - **Training examples:** 3,000 - **Training steps:** 60 - **Gradient accumulation:** 4 - **Learning rate:** 1e-4 - **Optimizer:** AdamW - **Scheduler:** OneCycleLR - **Weight decay:** 0.0 - **Max context characters:** 2,500 - **Canvas length:** 256 - **Dataset:** `qiaojin/PubMedQA` #### Speeds, Sizes, Times Training was run on a RunPod H100 notebook. Training logs from the saved notebook: ```text step 20/60 | loss 0.0019 | 43s step 40/60 | loss 0.0003 | 85s step 60/60 | loss 0.0001 | 126s ``` Approximate training time: ```text 126 seconds for 60 steps ``` ## Evaluation ### Testing Data, Factors & Metrics #### Testing Data Evaluation used the `pqa_labeled` subset of `qiaojin/PubMedQA`. The reported run used: - **Evaluation examples:** 50 - **Denoising steps:** 16 - **Metric:** Accuracy #### Factors The evaluation was not disaggregated by biomedical topic, article type, answer class, or question type. Results should be treated as a small sanity-check evaluation, not a full benchmark. #### Metrics Accuracy was used because PubMedQA final decisions are discrete labels: ```text yes / no / maybe ``` A prediction was counted as correct if the cleaned model output matched the gold `final_decision`. ### Results | Setting | Accuracy | Correct / Total | | --- | --- | --- | | Before fine-tuning | 0.60 | 30 / 50 | | After fine-tuning | 0.80 | 40 / 50 | | Improvement | +0.20 | +10 / 50 | #### Summary In the saved RunPod H100 notebook run, the model improved from **60% accuracy before fine-tuning** to **80% accuracy after fine-tuning** on a 50-example PubMedQA evaluation sample. This is a **+20 percentage point improvement**. The result shows that the model can quickly adapt to the PubMedQA answer format. However, this is a small evaluation and should not be interpreted as a clinically meaningful benchmark. ## Model Examination No detailed interpretability or model examination was performed. ## Environmental Impact Carbon emissions were not measured for this run. - **Hardware Type:** NVIDIA H100 80GB HBM3 - **Hours used:** Approximately 0.04 hours for the 60-step training loop, excluding setup, model loading, and evaluation - **Cloud Provider:** RunPod - **Compute Region:** Not recorded - **Carbon Emitted:** Not measured ## Technical Specifications ### Model Architecture and Objective The base model is DiffusionGemma 26B-A4B IT, a diffusion-style language model. The fine-tuning used LoRA adapters. The training objective followed a block-diffusion setup: 1. Encode the target answer into the model canvas. 2. Randomly corrupt answer tokens. 3. Train the model to reconstruct the clean answer. 4. Apply loss only over the target answer tokens. ### Compute Infrastructure #### Hardware - NVIDIA H100 80GB HBM3 - Reported GPU memory: approximately 85 GB total #### Software - Python - PyTorch 2.10.0+cu128 - Transformers - Unsloth - Unsloth Zoo - PEFT - Datasets - RunPod Jupyter Notebook ## Citation If you use this model, please cite the original PubMedQA dataset and DiffusionGemma base model. **PubMedQA:** ```bibtex @inproceedings{jin2019pubmedqa, title={PubMedQA: A Dataset for Biomedical Research Question Answering}, author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William W. and Lu, Xinghua}, booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing}, year={2019} } ``` ## Glossary - **PubMedQA:** A biomedical question-answering dataset based on PubMed abstracts. - **LoRA:** Low-Rank Adaptation, a parameter-efficient fine-tuning method. - **DiffusionGemma:** A diffusion-style language model. - **Denoising steps:** Iterative generation steps used by diffusion models. - **Accuracy:** Percentage of predictions matching the gold label. ## More Information This model was created as a simple fine-tuning experiment for adapting DiffusionGemma to a medical QA dataset. The task is intentionally simple: ```text Biomedical context + question → yes / no / maybe ``` For a stronger medical QA model, future versions should train on both: ```text Decision: yes/no/maybe Explanation: short evidence-based explanation ``` ## Model Card Authors - kingabzpro ## Model Card Contact For questions, contact the model repository owner on Hugging Face.