|
|
--- |
|
|
library_name: peft |
|
|
base_model: microsoft/phi-2 |
|
|
tags: |
|
|
- biomedical |
|
|
- summarization |
|
|
- lay-summary |
|
|
- healthcare |
|
|
- nlp |
|
|
- fine-tuned |
|
|
- lora |
|
|
- peft |
|
|
- elife |
|
|
- plos |
|
|
- medical-text |
|
|
language: |
|
|
- en |
|
|
license: mit |
|
|
metrics: |
|
|
- rouge |
|
|
- bertscore |
|
|
- readability |
|
|
datasets: |
|
|
- elife |
|
|
- plos |
|
|
pipeline_tag: text2text-generation |
|
|
--- |
|
|
|
|
|
# Phi-2 BioLaySum: Biomedical Lay Summarization Model 🏆 |
|
|
|
|
|
## 📖 Model Overview |
|
|
|
|
|
**Phi-2 BioLaySum** is a **champion model** that emerged as the most efficient and highest-performing solution for generating lay summaries of biomedical articles. This model converts complex medical research into easily understandable summaries for the general public, significantly enhancing accessibility to scientific literature. |
|
|
|
|
|
**🥇 Key Achievement**: This model **outperformed** T5-Base, T5-Large, FlanT5-Base, BioGPT, and Falconsi-Medical_summarisation across all evaluation dimensions (relevance, readability, and factuality) while maintaining optimal computational efficiency. |
|
|
|
|
|
## 🎯 Model Purpose |
|
|
|
|
|
This model addresses the critical need to bridge the gap between complex biomedical research and public health literacy by: |
|
|
- Converting medical articles into patient-friendly summaries |
|
|
- Supporting healthcare communication between professionals and patients |
|
|
- Enhancing public access to biomedical research findings |
|
|
- Enabling better-informed health decisions by the general public |
|
|
|
|
|
## 🏗️ Model Architecture |
|
|
|
|
|
- **Base Model**: microsoft/phi-2 |
|
|
- **Fine-tuning Technique**: LoRA (Low-Rank Adaptation) + PEFT (Parameter Efficient Fine-tuning) |
|
|
- **Model Type**: Text-to-Text Generation (Summarization) |
|
|
- **Language**: English |
|
|
- **Domain**: Biomedical/Healthcare |
|
|
|
|
|
## 📊 Performance Highlights |
|
|
|
|
|
### Why Phi-2 is the Champion Model: |
|
|
- ✅ **Superior Performance**: Best scores across relevance, readability, and factuality metrics |
|
|
- ✅ **Resource Efficiency**: Optimal performance-to-resource ratio |
|
|
- ✅ **Compact Size**: Most efficient in terms of model size and computational requirements |
|
|
- ✅ **Cost-Effective**: Best balance of quality and computational cost |
|
|
|
|
|
### Evaluation Results: |
|
|
- **Relevance**: Measured using ROUGE (1, 2, L) and BERTScore |
|
|
- **Readability**: Assessed via Flesch-Kincaid Grade Level (FKGL) and Dale-Chall Readability Score (DCRS) |
|
|
- **Factuality**: Verified using BARTScore and factual consistency checks |
|
|
|
|
|
## 🚀 Quick Start |
|
|
|
|
|
### Loading the Model |
|
|
|
|
|
```python |
|
|
from peft import PeftModel, PeftConfig |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
import torch |
|
|
|
|
|
# Load the base model and tokenizer |
|
|
base_model_name = "microsoft/phi-2" |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
base_model_name, |
|
|
torch_dtype=torch.float16, |
|
|
device_map="auto" |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained(base_model_name) |
|
|
|
|
|
# Load the fine-tuned adapter |
|
|
model = PeftModel.from_pretrained(model, "sank29mane/phi-2-biolaysum") |
|
|
|
|
|
# Set padding token |
|
|
if tokenizer.pad_token is None: |
|
|
tokenizer.pad_token = tokenizer.eos_token |
|
|
``` |
|
|
|
|
|
### Generating Lay Summaries |
|
|
|
|
|
```python |
|
|
def generate_lay_summary(medical_text, max_length=150): |
|
|
# Prepare input |
|
|
prompt = f"Summarize the following medical text for a general audience: {medical_text}" |
|
|
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512) |
|
|
|
|
|
# Generate summary |
|
|
with torch.no_grad(): |
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_length=max_length, |
|
|
temperature=0.7, |
|
|
do_sample=True, |
|
|
pad_token_id=tokenizer.eos_token_id |
|
|
) |
|
|
|
|
|
# Decode and return |
|
|
summary = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
return summary.split(":")[-1].strip() # Extract generated part |
|
|
|
|
|
# Example usage |
|
|
medical_text = """ |
|
|
The study investigated the efficacy of novel therapeutic interventions |
|
|
in cardiovascular disease management through randomized controlled trials... |
|
|
""" |
|
|
|
|
|
lay_summary = generate_lay_summary(medical_text) |
|
|
print(f"Lay Summary: {lay_summary}") |
|
|
``` |
|
|
|
|
|
## 📚 Training Details |
|
|
|
|
|
### Training Data |
|
|
- **eLife Dataset**: Open-access biomedical research articles with lay summaries |
|
|
- **PLOS Dataset**: Public Library of Science biomedical publications |
|
|
- **Data Processing**: Advanced preprocessing for optimal model performance |
|
|
|
|
|
### Training Configuration |
|
|
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation) with PEFT |
|
|
- **Base Model**: microsoft/phi-2 |
|
|
- **Training Framework**: PyTorch + Hugging Face Transformers |
|
|
- **Optimization**: Parameter-efficient approach reducing computational requirements |
|
|
|
|
|
### Training Advantages |
|
|
- **Efficiency**: LoRA reduces trainable parameters while maintaining performance |
|
|
- **Resource-Friendly**: PEFT enables high-quality fine-tuning with limited resources |
|
|
- **Stability**: Advanced techniques ensure robust model behavior |
|
|
|
|
|
## 📈 Comparative Analysis |
|
|
|
|
|
### Models Compared: |
|
|
1. **T5-Base** - Text-to-Text Transfer Transformer (Base) |
|
|
2. **T5-Large** - Text-to-Text Transfer Transformer (Large) |
|
|
3. **FlanT5-Base** - Instruction-tuned T5 model |
|
|
4. **BioGPT** - Biomedical domain-specific GPT |
|
|
5. **Phi-2** - Microsoft's efficient language model (**Winner**) |
|
|
6. **Falconsi-Medical_summarisation** - Specialized medical summarization model |
|
|
|
|
|
### Key Findings: |
|
|
- **Phi-2 outperformed all competitors** in comprehensive evaluation |
|
|
- **Domain-specific models** (BioGPT, Falconsi) showed advantages over general T5 models |
|
|
- **Parameter efficiency** of Phi-2 provided superior cost-effectiveness |
|
|
- **Smaller models** can achieve better performance with proper fine-tuning |
|
|
|
|
|
## 🎯 Use Cases |
|
|
|
|
|
### Healthcare Applications: |
|
|
- **Patient Education**: Convert research findings into understandable format |
|
|
- **Medical Communication**: Support doctor-patient conversations |
|
|
- **Health Journalism**: Assist science writers and health reporters |
|
|
- **Educational Materials**: Create teaching resources for health education |
|
|
- **Policy Support**: Provide accessible summaries for health policy decisions |
|
|
|
|
|
### Target Audiences: |
|
|
- Healthcare professionals seeking patient communication tools |
|
|
- Patients and families researching medical conditions |
|
|
- Health educators and trainers |
|
|
- Medical journalists and science communicators |
|
|
- Public health policy makers |
|
|
|
|
|
## ⚡ Performance Metrics |
|
|
|
|
|
### Evaluation Framework: |
|
|
- **ROUGE Scores**: Overlap-based relevance assessment |
|
|
- **BERTScore**: Semantic similarity evaluation |
|
|
- **Readability Metrics**: FKGL and DCRS for accessibility |
|
|
- **Factual Consistency**: BARTScore for accuracy verification |
|
|
|
|
|
### Resource Efficiency: |
|
|
- **Model Size**: Compact and deployment-friendly |
|
|
- **Inference Speed**: Fast generation suitable for real-time applications |
|
|
- **Memory Usage**: Optimized for various computational environments |
|
|
- **Cost Effectiveness**: Best performance per computational dollar |
|
|
|
|
|
## 🔧 Technical Specifications |
|
|
|
|
|
### Model Details: |
|
|
- **Architecture**: Transformer-based with LoRA adaptation |
|
|
- **Parameters**: Base Phi-2 + efficient LoRA adapters |
|
|
- **Precision**: Mixed precision training for efficiency |
|
|
- **Framework**: PyTorch with Hugging Face ecosystem |
|
|
|
|
|
### System Requirements: |
|
|
- **Minimum GPU**: 4GB VRAM for inference |
|
|
- **Recommended**: 8GB+ VRAM for optimal performance |
|
|
- **CPU**: Compatible with CPU inference (slower) |
|
|
- **Dependencies**: transformers, peft, torch |
|
|
|
|
|
## 📖 Research Impact |
|
|
|
|
|
This model contributes to: |
|
|
- **Democratizing Medical Knowledge**: Making research accessible to all |
|
|
- **Advancing Healthcare NLP**: Pushing boundaries of medical text processing |
|
|
- **Resource-Efficient AI**: Demonstrating effective use of LoRA and PEFT |
|
|
- **Evaluation Methodology**: Comprehensive framework for summarization assessment |
|
|
|
|
|
## 📄 License & Citation |
|
|
|
|
|
### License |
|
|
This model is released under the **MIT License**, promoting open research and development. |
|
|
|
|
|
### Citation |
|
|
If you use this model in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{mane2024phi2biolaysum, |
|
|
title={Phi-2 BioLaySum: Resource-Efficient Biomedical Lay Summarization using LoRA and PEFT}, |
|
|
author={Mane, Sanket}, |
|
|
year={2024}, |
|
|
publisher={Hugging Face}, |
|
|
url={https://huggingface.co/sank29mane/phi-2-biolaysum} |
|
|
} |
|
|
``` |
|
|
|
|
|
## 🔗 Related Resources |
|
|
|
|
|
- **GitHub Repository**: [lays-bio-summery](https://github.com/sank29mane/lays-bio-summery) - Complete training code and evaluation |
|
|
- **Base Model**: [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) |
|
|
- **Research Paper**: [Detailed methodology and results](https://github.com/sank29mane/lays-bio-summery) |
|
|
|
|
|
## 👨💻 Author |
|
|
|
|
|
**Sanket Mane** - [@sank29mane](https://github.com/sank29mane) |
|
|
*Researcher in Biomedical NLP and Efficient Language Models* |
|
|
|
|
|
## 📞 Contact & Support |
|
|
|
|
|
- **GitHub Issues**: [Create an issue](https://github.com/sank29mane/lays-bio-summery/issues) |
|
|
- **Model Issues**: Use the Community tab above |
|
|
- **Research Collaborations**: Through GitHub profile |
|
|
|
|
|
## 🚨 Limitations & Considerations |
|
|
|
|
|
### Current Limitations: |
|
|
- **Language**: Currently optimized for English biomedical text |
|
|
- **Domain**: Focused on general biomedical research (not clinical notes) |
|
|
- **Length**: Optimized for article-length inputs, may vary with very short/long texts |
|
|
|
|
|
### Recommended Use: |
|
|
- Use for biomedical research article summarization |
|
|
- Validate outputs for critical healthcare decisions |
|
|
- Consider human review for patient-facing applications |
|
|
|
|
|
## 🔄 Model Updates |
|
|
|
|
|
- **v1.0**: Initial release with LoRA+PEFT fine-tuning |
|
|
- **Future**: Planned improvements for multi-language support and clinical text adaptation |
|
|
|
|
|
--- |
|
|
|
|
|
### Framework Versions |
|
|
- **PEFT**: 0.7.2.dev0 |
|
|
- **Transformers**: Compatible with latest versions |
|
|
- **PyTorch**: 1.12+ |
|
|
|
|
|
⭐ **Star this model if you find it useful for your biomedical NLP research!** ⭐ |
|
|
|
