PentaMind

PentaMind predicts global cognition from a scanned drawing of intersecting pentagons (the pentagon copying item of the MMSE). It is an ensemble of 10 VGG19-bn models, trained on 13,777 drawings from 3,111 older adults in the ROS, MAP and MARS cohorts of the Rush Alzheimer's Disease Center (RADC).

• Code and usage: github.com/stasaki/PentaMind
• Paper: Tasaki S, Kim N, Truty T, Zhang A, Buchman AS, Lamar M, Bennett DA. Explainable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons. npj Digital Medicine 6, 157 (2023). doi:10.1038/s41746-023-00904-w

Model details

Task	Regression: one image -> one number (predicted global cognition, higher = better)
Architecture	torchvision vgg19_bn, ImageNet start, classifier[6] replaced by Linear(4096, 1)
Files	models/vgg19_bn-1_model.pth ... models/vgg19_bn-10_model.pth (PyTorch state_dict, fp32, 558 MB each)
Ensemble	Average the 10 model outputs. The 10 models come from 10-fold cross-validation (each trained on 9/10 of the participants)
Input	RGB, 224 x 224, ImageNet normalization (mean 0.485/0.456/0.406, std 0.229/0.224/0.225)
Output scale	ROS/MAP/MARS global cognition: composite of 19 cognitive tests on a z-score scale (0 = about the cohort average)
Framework	Trained with PyTorch 1.10; loads with current PyTorch (tested with 2.x, CPU and GPU)

How to use

The easiest way is the script in the GitHub repository. It downloads these weights, preprocesses the images and averages the 10 models:

git clone https://github.com/stasaki/PentaMind.git && cd PentaMind
pip install -r requirements.txt
python pentamind_predict.py --images my_images/ --out scores.csv          # add --crop for whole-page scans

Minimal Python example (one image, 10 models):

import torch, torchvision
from torch import nn
from PIL import Image
from huggingface_hub import hf_hub_download

net = torchvision.models.vgg19_bn(weights=None)
net.classifier[6] = nn.Linear(4096, 1)
net.eval()

prep = torchvision.transforms.Compose([
    torchvision.transforms.Resize((224, 224)),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
x = prep(Image.open("pentagon.png").convert("RGB")).unsqueeze(0)  # drawing on white, square image

scores = []
for i in range(1, 11):
    path = hf_hub_download("stasaking/pentamind", f"models/vgg19_bn-{i}_model.pth")
    net.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
    with torch.no_grad():
        scores.append(net(x).item())
print("PentaMind score:", sum(scores) / len(scores))

Input images

Training images were scanned pencil-and-paper drawings. The pentagon region was cut out, padded with white to a 500 x 500 square and resized to 224 x 224. So the model expects dark lines on a white background, the drawing centered and filling most of a square image, and nothing else (no text, page borders or signatures). The GitHub script's --crop option does this for whole-page scans.

In the paper, each image was also scored 30 times with small random changes (test-time augmentation) and the results were averaged. Use --n_tta 30 in the script to do the same.

Training data

• 13,777 drawings from 3,111 participants: ROS (1,093 participants), MAP (1,508), MARS (510).
• Label: global cognition at the same research visit as the drawing (19 tests; the MMSE and the pentagon item are not part of it).
• Split by participant: 80% training, 10% validation, 10% test in model selection; 10-fold cross-validation for the 10 released models.
• The data are not public. They can be requested from the RADC Resource Sharing Hub under a data use agreement.

Performance (from the paper)

• Test set: Spearman correlation 0.44 with global cognition.
• Out-of-fold: explained 23.3% of the variance in global cognition, 1.9 times more than the usual pass/fail rating of the drawing.
• Separating dementia from no cognitive impairment: AUC 0.77 (pass/fail rating: 0.67).
• The score was also related to motor function and to several brain pathologies.

Intended use

• Research on cognition and drawing, for example as a cognitive variable in studies of aging, or as a feature extractor for other drawing tasks.
• Not for clinical diagnosis or decisions about individuals.

Limitations and cautions

• Data leakage on RADC data. Every ROS/MAP/MARS drawing in the training data was used to train 9 of the 10 models. Scores on those drawings will look better than they really are. Do not use them to measure performance.
• Training population: older adults in the US, mostly White (MARS: African American), one test form and scanning setup. Performance in other populations, forms or scanners is unknown.
• Other drawings (clocks, cubes, etc.) are out of domain. The score on them is usually only weakly related to cognition. Training a simple model (for example ridge regression) on the image features (see --features in the GitHub script) on your own labels usually works better.
• Digital pen data: the model only sees the final image (no timing, pressure or stroke order). Render strokes as black lines on white, with a realistic line width (very thin lines lowered the score in the paper's simulation), and crop to the drawing.
• Modest accuracy: the score explains about a quarter of the variance in cognition and is pulled toward the average.

License

The weights are licensed under CC BY-NC-SA 4.0 (attribution, non-commercial, share-alike), because they were trained on ROS/MAP/MARS data. The code on GitHub is BSD 3-Clause.

Citation

@article{tasaki2023pentamind,
  title   = {Explainable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons},
  author  = {Tasaki, Shinya and Kim, Namhee and Truty, Tim and Zhang, Ada and Buchman, Aron S. and Lamar, Melissa and Bennett, David A.},
  journal = {npj Digital Medicine},
  volume  = {6},
  pages   = {157},
  year    = {2023},
  doi     = {10.1038/s41746-023-00904-w}
}

Acknowledgments

We thank all the participants of the ROS, MAP and MARS studies.