README.md

---
license: cc-by-nc-4.0
library_name: transformers
tags:
  - dermatology
  - medical-imaging
  - embeddings
  - image-feature-extraction
  - clip
pipeline_tag: image-feature-extraction
---

# A Global Atlas of Digital Dermatology to Map Innovation and Disparities

![SkinMap method](https://fabiangroeger96.github.io/assets/img/projects/skinmap/Method.svg)

SkinMap is a dermatology image/text embedding model. It fuses **12 frozen teacher
encoders** (multi-modal and self-supervised models) through a trained projector
into a single **1024-d shared image/text space**, that can be used to infer
missing metadata, solve downstream tasks, and get relational information about the
samples.

**Compute.** This is the faithful full ensemble: 12 backbones load into memory at
once and need roughly **16 GB of GPU VRAM**. It runs on CPU but slowly.

## Usage

```python
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Digital-Dermatology/SkinMap",
    trust_remote_code=True,
    device="cuda",            # "cpu" also works
)

# 1) Encode an image into the 1024-d SkinMap space (L2-normalized)
img_emb = model.encode_image("lesion.jpg")            # np.ndarray, shape (1024,)

# 2) Encode a text query into the same space
txt_emb = model.encode_text("melanoma on the back")   # np.ndarray, shape (1024,)

# 3) Predict clinical metadata from the image
meta = model.predict_meta("lesion.jpg")
# {'fitzpatrick': 3, 'fitzpatrick_grouped': '3-4', 'age': 42.1,
#  'gender': 'female', 'origin': ..., 'body_region': ..., ...}

import numpy as np
sim = float(np.dot(img_emb, txt_emb))                 # cosine in the shared space
```

`encode_image` accepts a file path, raw bytes, or a `PIL.Image`. `encode_text`
expands the query into clinical prompt templates and averages the results; pass
`templates=["{}"]` to encode the raw text as-is.

### Metadata prediction

`predict_meta` applies bundled linear probes to the SkinMap embedding to estimate
Fitzpatrick skin type, age, sex, geographic origin, body region and more. By
default it uses **global** probes, which reproduce the demographic estimates
reported in the paper. If you know the imaging modality, pass it to switch the
Fitzpatrick prediction to a **modality-specific** probe (more accurate when the
modality is known); the other attributes stay global:

```python
meta = model.predict_meta("lesion.jpg", modality="dermoscopy")  # clinical | dermoscopy | TBP
```

`predict_meta` also accepts a precomputed 1024-d vector instead of an image, and a
subset via `attributes=[...]`.

### Nearest-neighbor search (optional)

If this repo was built with an atlas index bundled, you can retrieve neighbors
directly:

```python
indices, distances = model.search("lesion.jpg", k=10)
```

Without a bundled index, `search()` raises a clear error. Encoding still works.

## What's inside

| Component | Role |
|---|---|
| 9 multi-modal teachers (CLIP & SigLIP; ViT-L/14 & ViT-B/32) | image + text features |
| 3 self-supervised teachers (DINO, iBOT, MAE; ViT-B/16) | dermatology image features |
| Per-teacher whitening (`whitening_stats.npz`) | decorrelate before fusion |
| Trained projector (`projector_model.pth`) | fuse to 1024-d image/text heads |
| Metadata probes (`probes/`) | linear probes for `predict_meta` |

## Licensing

SkinMap is released under the Creative Commons Attribution-NonCommercial 4.0
International license, for research use only. **Not a medical device. Not for
clinical decision-making.**