File size: 2,547 Bytes
8d1d923 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
base_model: meta-llama/Llama-3.2-3B-Instruct
library_name: peft
pipeline_tag: text-generation
license: llama3.2
tags:
- llama
- lora
- peft
- qlora
- biomedical
- healthcare
- pubmed
- medical
- question-answering
---
# HealthGPT-LoRA
HealthGPT-LoRA is a biomedical question-answering model built by fine-tuning **Meta Llama 3.2 3B Instruct** using **QLoRA (PEFT)** on the **PubMedQA** dataset.
This repository contains **only the LoRA adapter**, which can be loaded on top of the original Llama 3.2 3B Instruct model.
---
# Model Details
- **Base Model:** Meta Llama 3.2 3B Instruct
- **Fine-tuning Method:** QLoRA (PEFT)
- **Task:** Biomedical Question Answering
- **Framework:** Transformers + PEFT
- **Quantization:** 4-bit NF4 (BitsAndBytes)
- **Precision:** BF16 Mixed Precision
---
# Dataset
The model is trained on the **PubMedQA** dataset containing biomedical question-answer pairs.
### Current Training Progress
- **Training Samples:** ~150,000
- **Dataset Completion:** Approximately **75%**
- Training is currently in progress, with plans to continue training on the remaining dataset and further improve performance.
---
# Current Evaluation Results
| Metric | Score |
|--------|------:|
| Accuracy | **93.4%** |
| Precision | **98.1%** |
| Recall | **93.4%** |
| F1 Score | **95.7%** |
The current model achieves approximately **12% higher accuracy** than the base Llama 3.2 3B Instruct model on the evaluation dataset.
---
# Planned Improvements
Upcoming milestones include:
- Retrieval-Augmented Generation (RAG)
- FAISS Vector Database
- Multi-source Medical Knowledge Retrieval
- FastAPI Backend
- Docker Deployment
- CI/CD Pipeline
- Web-based Interface
---
# Usage
```python
from transformers import AutoModelForCausalLM
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B-Instruct"
)
model = PeftModel.from_pretrained(
base_model,
"llmithull/HealthGPT-LoRA"
)
```
---
# Repository Contents
This repository includes:
- LoRA Adapter Weights
- Adapter Configuration
- Training Configuration
The original Llama 3.2 model is **not included** and must be downloaded separately from Hugging Face.
---
# Project Status
🚧 **Active Development**
HealthGPT is an ongoing project focused on building a production-ready biomedical AI assistant. The current release represents approximately **75% of the planned fine-tuning process**, with Retrieval-Augmented Generation (RAG) and deployment planned in future updates. |