license: mit
language:
- en
tags:
- pytorch
- transformer
- gpt
- medical-ai
- clinical-nlp
- biomedical-nlp
- language-model
- healthcare
pipeline_tag: text-generation
🩺 MediGPT: A Domain-Specific Clinical Small Language Model
MediGPT is a decoder-only GPT-style Transformer trained entirely from scratch on medical and biomedical text corpora.
Unlike models that rely on large-scale pretraining followed by fine-tuning, MediGPT was developed as a research-oriented educational project to explore domain-specific language modeling, transformer architectures, and clinical text generation.
🚀 Key Features
- Built entirely from scratch using PyTorch
- Decoder-only GPT architecture
- Trained on medical and biomedical datasets
- GPT-2 BPE tokenization via tiktoken
- End-to-end implementation including:
- Data preprocessing
- Corpus analysis
- Transformer implementation
- Training pipeline
- Evaluation metrics
- Text generation
🏗️ Model Architecture
| Component | Value |
|---|---|
| Architecture | GPT Decoder |
| Layers | 8 |
| Attention Heads | 8 |
| Hidden Size | 512 |
| Context Length | 512 |
| Dropout | 0.1 |
| Tokenizer | GPT-2 (tiktoken) |
| Framework | PyTorch |
Approximate Parameter Count: ~90M
📚 Training Data
MediGPT was trained on a combination of medical datasets:
MedQuAD
Medical question-answer pairs covering diseases, symptoms, diagnostics, treatments, and healthcare information.
PubMedQA
Biomedical question-answer data derived from scientific literature and PubMed abstracts.
The resulting corpus exposes the model to:
- Clinical terminology
- Biomedical vocabulary
- Medical QA patterns
- Research-style writing
- Healthcare discourse
🎯 Training Objective
The model was trained using autoregressive next-token prediction.
Given a sequence such as:
Symptoms of diabetes include ...
the model learns to predict the most likely next token.
📈 Evaluation
Evaluation included:
- Validation Loss
- Perplexity
- Corpus Statistics
- Attention Analysis
- Qualitative Text Generation
Results indicate successful learning of:
- Clinical vocabulary
- Biomedical terminology
- Medical writing style
- Research-oriented sentence structures
🔬 Example Prompt
Input:
Symptoms of diabetes include
Generated continuation:
Symptoms of diabetes include increased thirst, frequent urination, fatigue, blurred vision and other complications associated with impaired glucose regulation.
⚠️ Limitations
- Small-scale model compared to modern foundation models
- Limited medical reasoning capability
- May generate inaccurate information
- Not suitable for clinical deployment
- Intended primarily for research and educational purposes
🩺 Medical Disclaimer
This model is not a medical device and must not be used for:
- Medical diagnosis
- Treatment recommendations
- Clinical decision making
- Emergency healthcare situations
All outputs should be verified by qualified healthcare professionals.
👨💻 Author
Rishabh Shenoy
Developed as a research-oriented educational project exploring domain-specific language model development and biomedical NLP.
📜 Citation
If you use this work, please cite the repository and model page.