File size: 4,536 Bytes
f25b064 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | ---
library_name: pytorch
pipeline_tag: tabular-classification
tags:
- pytorch
- transformer
- electronic-health-records
- ehr
- healthcare
- multi-label-classification
- next-event-prediction
---
# EHR Transformer Decoder
A causal Transformer for learning from longitudinal electronic health record (EHR) event sequences and predicting incident clinical conditions over a five-year horizon.
This repository contains two PyTorch checkpoints:
| File | Contents |
| --- | --- |
| `pretrained_decoder.pt` | Decoder pretrained to predict the next clinical event code |
| `best_model.pt` | Full decoder and multi-label classification head selected by validation mean average precision |
> **Research use only.** This model is not a medical device or clinical decision-support system. Its outputs must not be used to diagnose, treat, or make decisions about patients.
## Model description
The model processes a patient's coded clinical history in chronological order. Each event combines:
- A clinical code embedding
- An event-source/type embedding
- A learned position embedding
- Three continuous time features
- Patient gender and race embeddings
The time features represent days before the prediction anchor, age at the event, and time since the previous event. A causal Transformer learns the sequence representation. The fine-tuned model pools the last non-padding state and produces one probability per target condition.
Supported event sources are conditions, medications, procedures, observations, encounters, care plans, and immunizations.
## Checkpoint format
Both files are raw PyTorch `state_dict` checkpoints saved with `torch.save(model.state_dict(), ...)`. They are **not** Hugging Face Transformers `AutoModel` checkpoints and cannot be loaded with `AutoModel.from_pretrained()` or the hosted inference widget.
Inference requires the model classes and preprocessing pipeline from the source project. It also requires the exact vocabulary, ordered target-condition list, and architecture used during training.
## Installation
Clone the source project, then install its dependencies:
```bash
git clone <git@github.com:Salma-Jamal/Forecasting_Future_Conditions.git>
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch pandas numpy scikit-learn tqdm huggingface_hub
```
## Download the checkpoints
`SalmaJamal/Forecasting_Future_Conditions`
### Python
```python
from huggingface_hub import hf_hub_download
repo_id = "SalmaJamal/Forecasting_Future_Conditions"
best_model_path = hf_hub_download(
repo_id=repo_id,
filename="best_model.pt",
)
pretrained_decoder_path = hf_hub_download(
repo_id=repo_id,
filename="pretrained_decoder.pt",
)
print(best_model_path)
print(pretrained_decoder_path)
```
### Command line
```bash
hf download SalmaJamal/Forecasting_Future_Conditions \
best_model.pt pretrained_decoder.pt \
--local-dir ./checkpoints
```
## Use the pretrained decoder
The source project can rebuild the vocabulary from the training split and use `pretrained_decoder.pt` to initialize fine-tuning:
```bash
python run.py \
--data-dir ./data \
--output-dir ./outputs/finetune \
--skip-pretrain \
--pretrain-ckpt ./checkpoints/pretrained_decoder.pt
```
## Expected input data
The preprocessing code expects a directory containing:
```text
data/
├── patient_splits.csv
├── target_conditions.csv
├── test_anchors.csv # optional
├── train_val/
│ ├── patients.csv
│ ├── encounters.csv
│ ├── conditions.csv
│ ├── observations.csv
│ ├── medications.csv
│ ├── procedures.csv
│ ├── immunizations.csv
│ └── careplans.csv
└── test/
└── ...same table names...
```
`patient_splits.csv` requires `Id` and `split` columns. `target_conditions.csv` requires a `CODE` column. Event tables use `PATIENT`, `CODE`, and their source-specific date column. See the source project's README for the complete schema.
## Training objective
Training uses two stages:
1. **Causal pretraining:** next-event code prediction.
2. **Multi-label fine-tuning:** prediction of target conditions that first occur within five years after the anchor.
The default fine-tuning setup uses class-weighted focal loss, an auxiliary next-event loss, multi-anchor training augmentation, AdamW, a one-cycle learning-rate schedule, gradient clipping, and early stopping on validation mean average precision. |