codehance/distilgpt2-medical-pharma

DistilGPT-2 Medical Pharmaceutical Autocomplete

Model Description

This is a distilled GPT-2 model fine-tuned for pharmaceutical autocomplete. It suggests drug names and medical terminology based on clinical context.

Key Features:

• 34% smaller than base fine-tuned model (81,912,576 parameters)
• 45% faster inference (347.9ms per generation)
• Specialized in pharmaceutical vocabulary

Training Process

Stage 1: Fine-Tuning

• Base model: GPT-2 (124M parameters)
• Dataset: Medical Q&A (medmcqa) - 4,500 training examples
• Training: 3 epochs
• Final perplexity: 23.61

Stage 2: Knowledge Distillation

• Teacher: Fine-tuned GPT-2
• Student: DistilGPT-2
• Training: 2 epochs
• Compression: 34.2% size reduction

Performance

Metric	Value
Parameters	81,912,576
Perplexity	44.07
Inference Speed	347.9ms
Quality Retained	53.6%

Usage

from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained("codehance/distilgpt2-medical-pharma")
tokenizer = GPT2Tokenizer.from_pretrained("codehance/distilgpt2-medical-pharma")

# Generate pharmaceutical suggestions
prompt = "The patient should take"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=30, num_return_sequences=3)

for output in outputs:
    print(tokenizer.decode(output, skip_special_tokens=True))

Intended Use

Primary Use Cases:

• Pharmaceutical autocomplete systems
• Medical documentation assistance
• Clinical note-taking tools
• Drug name suggestion

Limitations:

• Not a substitute for medical advice
• May suggest incorrect drugs - always verify with qualified professionals
• Trained on medical exam questions, not real prescriptions
• English language only

Training Data

• Source: MedMCQA dataset (Indian medical entrance exam questions)
• Size: 4,500 training examples
• Content: Medical questions with pharmaceutical terminology

Ethical Considerations

⚠️ Important: This model is for autocomplete assistance only. It should NOT be used as the sole basis for medical decisions. Always verify suggestions with qualified healthcare professionals.

Model Card Authors

Created as part of a pharmaceutical autocomplete system tutorial demonstrating transfer learning, fine-tuning, and knowledge distillation.

Citation

@misc{distilgpt2-medical-pharma,
  author = {codehance},
  title = {DistilGPT-2 Medical Pharmaceutical Autocomplete},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/codehance/distilgpt2-medical-pharma}}
}