|
|
--- |
|
|
language: "en" |
|
|
tags: |
|
|
- biomedical |
|
|
- text-generation |
|
|
- BioGPT |
|
|
- fine-tuning |
|
|
license: "cc-by-4.0" |
|
|
datasets: |
|
|
- custom |
|
|
metrics: |
|
|
- perplexity |
|
|
- loss |
|
|
--- |
|
|
|
|
|
# TissueGPT: Fine-Tuned BioGPT for Tissue Engineering Text Generation |
|
|
|
|
|
## Model Description |
|
|
**TissueGPT** is a fine-tuned version of [BioGPT](https://huggingface.co/microsoft/BioGPT), specifically tailored for tissue engineering text generation tasks. By leveraging a dataset of biomedical research articles (titles, abstracts, and full texts), TissueGPT is designed to perform tasks such as: |
|
|
|
|
|
- Summarizing biomedical literature |
|
|
- Generating coherent biomedical text |
|
|
- Assisting with scientific writing in life sciences |
|
|
- Supporting research in tissue engineering, extracellular matrix (ECM) analysis, and related fields |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### First Round of Training |
|
|
The initial model was fine-tuned for **3 epochs**, focusing on general adaptation to the biomedical dataset. |
|
|
|
|
|
#### Hyperparameters |
|
|
- **Learning Rate**: 5e-5 |
|
|
- **Batch Size**: 8 |
|
|
- **Warmup Steps**: 500 |
|
|
- **Precision**: Mixed precision (`fp16`) |
|
|
- **Weight Decay**: 0.01 |
|
|
- **Number of Epochs**: 3 |
|
|
- **Save Checkpoints**: Every 10,000 steps, keeping the last 3 checkpoints |
|
|
|
|
|
#### Training and Validation Metrics |
|
|
| Epoch | Training Loss | Validation Loss | Perplexity | |
|
|
|-------|---------------|-----------------|------------| |
|
|
| 1 | 2.4752 | 2.4286 | 11.34 | |
|
|
| 2 | 2.3680 | 2.3708 | 10.70 | |
|
|
| 3 | 2.2954 | 2.3410 | 10.39 | |
|
|
|
|
|
--- |
|
|
|
|
|
### Second Round of Training |
|
|
To further improve performance, the model was fine-tuned for **2 additional epochs** with adjusted hyperparameters. |
|
|
|
|
|
#### Adjusted Hyperparameters |
|
|
- **Learning Rate**: 3e-5 (reduced for finer updates) |
|
|
- **Batch Size**: 64 (to utilize the GPU’s full memory) |
|
|
- **Precision**: `bf16` (optimized for NVIDIA A100) |
|
|
- **Save Checkpoints**: Every 20,000 steps |
|
|
|
|
|
#### Training and Validation Metrics |
|
|
| Epoch | Training Loss | Validation Loss | Perplexity | |
|
|
|-------|---------------|-----------------|------------| |
|
|
| 4 | 2.2396 | 2.2395 | 9.43 | |
|
|
| 5 | 2.2328 | 2.2328 | 9.32 | |
|
|
|
|
|
### Hardware Used |
|
|
- **GPU**: NVIDIA A100 80GB |
|
|
- **Framework**: PyTorch with Hugging Face Transformers library |
|
|
|
|
|
--- |
|
|
|
|
|
## Evaluation Metrics |
|
|
|
|
|
### Perplexity |
|
|
Perplexity is a key metric for evaluating language models, measuring how well the model predicts sequences of text. Lower perplexity indicates better predictive performance. |
|
|
|
|
|
- **First Round of Training**: Final perplexity = **10.39** |
|
|
- **Second Round of Training**: Final perplexity = **9.32** |
|
|
|
|
|
A lower perplexity indicates that the model generates more fluent and coherent text. |
|
|
|
|
|
### Gradient Norms |
|
|
- Tracked gradient stability during training. |
|
|
- Observed Range: **1.05–1.32**, indicating stable training. |
|
|
|
|
|
### Validation Loss |
|
|
- Decreasing validation loss across both rounds suggests effective generalization to unseen data. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Comparison |
|
|
|
|
|
| Metric | First Round | Second Round | |
|
|
|--------------------|-------------|--------------| |
|
|
| Final Validation Loss | 2.3410 | 2.2328 | |
|
|
| Final Perplexity | 10.39 | 9.32 | |
|
|
|
|
|
**Key Insights**: |
|
|
- Additional training epochs led to improved generalization and better predictive performance. |
|
|
- Perplexity improved by approximately 10% in the second round, demonstrating enhanced text fluency and coherence. |
|
|
|
|
|
--- |
|
|
|
|
|
## How to Use the Model |
|
|
|
|
|
### Install Dependencies |
|
|
Ensure you have `transformers` and `torch` installed: |
|
|
|
|
|
```bash |
|
|
pip install transformers torch |
|
|
``` |
|
|
### Load the Model |
|
|
|
|
|
``` python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model_name = "Saeed/TissueGPT" # Replace with the uploaded repo name |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_name) |
|
|
|
|
|
input_text = "The extracellular matrix plays a critical role in tissue engineering because" |
|
|
inputs = tokenizer(input_text, return_tensors="pt") |
|
|
|
|
|
output = model.generate(**inputs, max_length=50) |
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
---------- |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
- **Biomedical text generation and summarization** |
|
|
- **Assisting researchers, scientists, and medical professionals** |
|
|
- **Automated scientific writing** in domains like tissue engineering, and scaffold fabrication. |
|
|
|
|
|
---------- |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- The model is fine-tuned on biomedical literature and may not generalize well to non-biomedical domains. |
|
|
- Outputs should always be validated by experts for accuracy, especially in clinical or research-critical contexts. |
|
|
|
|
|
---------- |
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
- This model is intended for use in biomedical research and not for clinical diagnosis or patient care. |
|
|
- It may generate plausible-sounding but factually incorrect outputs (hallucinations). Always verify generated content. |
|
|
|
|
|
---------- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use **TissueGPT**, please cite the following: |
|
|
|
|
|
***The citation details will be provided shortly.*** |
|
|
## License |
|
|
|
|
|
Licensed under the **CC BY 4.0** License. |
|
|
## Contact |
|
|
|
|
|
For questions, issues, or collaboration opportunities, feel free to reach out at: |
|
|
|
|
|
- **Name**: Saeed Rafieyan |
|
|
- **Website**: Sraf.ir |
|
|
- **Email**: Raf.Biomed@gmail.com |
|
|
- **LinkedIn**: https://www.linkedin.com/in/saeed-rafieyan |
