README.md

---
license: other
license_name: dinov3-license
license_link: https://ai.meta.com/resources/models-and-libraries/dinov3-license/
datasets:
  - jilab/CellImageNet
library_name: pytorch
pipeline_tag: image-classification
tags:
  - biology
  - single-cell
  - cell-type-classification
  - microscopy
  - DAPI
  - vision-transformer
  - lora
  - dinov3
---

# MorphPT

**MorphPT** classifies cell types from **DAPI-stained nuclei**. Each cell is
described by two co-registered grayscale crops — a **2.5×** view (fine nuclear
morphology) and a **10×** view (broader tissue context) — encoded by a shared **DINOv3
ViT-B** backbone with **LoRA** adaptation and fused by a learnable **ScaleGate**
module. A coarse **router** predicts one of six broad morphology-informed
groups, and group **experts** resolve the fine cell type via soft routing.

Here **2.5×** and **10×** denote crop field-of-view *scales* (how much area
around the cell each crop spans), **not** microscope objective magnification:
the 2.5× crop is a tight view of the nucleus, while the 10× crop covers more
surrounding tissue.

- 💻 **Code:** [github.com/AnitaCao/MorphPT](https://github.com/AnitaCao/MorphPT)
- 🗂️ **CellImageNet benchmark dataset:** [huggingface.co/datasets/jilab/CellImageNet](https://huggingface.co/datasets/jilab/CellImageNet)
- 🤖 **MorphAgent (NL controller):** [github.com/ZBox1005/MorphAgent](https://github.com/ZBox1005/MorphAgent)
- 📄 **Paper:** "A visual foundation model for cell classification" — _citation forthcoming_

## Files

```
README.md                       # this card
LICENSE.md                      # DINOv3 License (required)
FINE_LABELS.md                  # fine cell types per coarse group (reference)
router/best.pt                  router/coarse_to_id.json     # 6-way coarse router
expert_Cancer/best.pt           expert_Cancer/class_to_idx.json
expert_Lymphoid/best.pt         expert_Lymphoid/class_to_idx.json
expert_Neuroglial/best.pt       expert_Neuroglial/class_to_idx.json
expert_Tissue_Vascular/best.pt  expert_Tissue_Vascular/class_to_idx.json
splits/                         # split spec + label maps (reproducibility)
```

Each checkpoint is self-contained and includes model weights, `args`, and label metadata needed for inference. The architecture is reconstructed from `args` at load time. The frozen DINOv3 ViT-B/16 backbone weights are embedded in each checkpoint.


## Label space

The released model covers **21 fine cell types** grouped into **6 broad
morphology-informed groups** (a human-tissue subset of CellImageNet):

| Group | Type of stage | Fine cell types |
|---|---|---|
| Cancer | expert | Colon / Liver / Lung / Ovary / Pancreas / Skin cancer cells |
| Lymphoid | expert | B cells, NK cells, T cells |
| Neuroglial | expert | Astrocytes, Microglia, Neurons, Oligodendrocytes |
| Tissue_Vascular | expert | Endothelial, Epithelial, Fibroblasts, Myeloid, Pericytes, Smooth muscle cells |
| Stromal | passthrough | Stromal cells |
| Stem_Progenitor | passthrough | Stem and progenitor cells |

Single-type groups (Stromal, Stem_Progenitor) are passthrough: the router label
maps directly to the cell type with no expert.

## Usage

Install the MorphPT package, then load the model and predict:

```bash
pip install git+https://github.com/AnitaCao/MorphPT.git
```

```python
from morphpt import MorphPTPredictor

model = MorphPTPredictor.from_pretrained("jilab/MorphPT")   # downloads the weights
out = model.predict_one("cell_2p5x.png", "cell_10x.png")
print(out["pred"], out["coarse_group"], round(out["confidence"], 3))
```

Or download the weights and run the example CLI:

```bash
hf download jilab/MorphPT --repo-type model --local-dir morphpt_weights
python examples/predict_cell.py --weights_dir morphpt_weights \
    --img_2p5x cell_2p5x.png --img_10x cell_10x.png
```

Inputs are paired grayscale DAPI crops of the same cell. Crops are resized to
224×224 and ImageNet-normalized internally.

Prediction uses soft routing. For multi-type groups, expert probabilities are
weighted by the router probability for that group. For passthrough groups, the
router probability is assigned directly to the corresponding fine label. The
final cell type is chosen by the highest fine-type score:

```
s(y) = Σ_{g in experts} q(g) p_g(y)  +  Σ_{g in passthrough} q(g) 1[y = y_g]
```

## Training data

Trained on a **human-tissue subset of CellImageNet**, subsampled per class to
mitigate class imbalance. CellImageNet is a large-scale single-cell DAPI image
database (~10M cells; 28 human + 14 mouse tissues) built from public 10x
Genomics Xenium data; the full dataset is released separately as a benchmark at
https://huggingface.co/datasets/jilab/CellImageNet.
The split specification used to train this model (seed, per-class subsample
targets, and label maps) is provided in this repository under `splits/` for
reproducibility. This checkpoint does **not** cover mouse tissues or cell types
outside the 21 listed above.

## Intended use and limitations

**Intended use.** MorphPT is intended for research use in fine-grained cell-type classification from DAPI nuclear-morphology crops in human tissue, including Xenium-style single-cell imagery. It may also be useful as a morphology-pretrained backbone for related research tasks.

**Not for clinical or diagnostic use.** This model is a research artifact, not a medical device, and should not be used for clinical diagnosis, treatment decisions, or other patient-care decisions.

**Scope and limitations.**
- Human tissue and DAPI input only. Mouse tissue, other stains, and other imaging modalities are unvalidated.
- Inputs must be correctly framed 2.5× and 10× crops of the same cell. Resizing fixes pixel size, not field of view. Mismatched crop scales may silently degrade predictions.
- The model was trained on a per-class-subsampled subset of CellImageNet. Rare cell types and out-of-distribution tissues may have weaker performance.
- Coarse routing is the main error source. Cells near coarse-group boundaries are likely to be less reliable.

## Related projects

**MorphAgent** is a natural-language controller that orchestrates cell
segmentation, MorphPT-based cell-type classification, and tumor–immune spatial
analysis through an interactive web interface —
[github.com/ZBox1005/MorphAgent](https://github.com/ZBox1005/MorphAgent) (MIT
License). MorphAgent trains no models of its own: it delegates cell-type
prediction to MorphPT and performs planning with a general-purpose LLM accessed
through a provider API (e.g. OpenAI GPT-4o or Anthropic Claude), so no model
weights are distributed with it. A versioned archival snapshot will be deposited
on Zenodo upon publication.

## Citation

```bibtex
@article{cao2026visual,
  title   = {A visual foundation model for cell classification},
  author  = {Cao, Ting and Zhuang, Haotian and Zhang, Boxuan and
             Pang, Zhiping P. and Tang, Ruixiang and Liu, Dongfang and
             Ji, Zhicheng},
  year    = {2026}
}
```

This work builds on DINOv3 (Meta AI) and the `timm` library; please also cite
DINOv3 and `timm` (Ross Wightman, *PyTorch Image Models*).

## License

This model is a **derivative work of Meta's DINOv3** — the DINOv3 ViT-B backbone
weights are embedded in every checkpoint — and is therefore distributed under
the **DINOv3 License**, not a standard permissive license. A copy of the DINOv3
License is included in this repository as **`LICENSE.md`**, copied from the
official `facebookresearch/dinov3` repository. Authoritative page:
<https://ai.meta.com/resources/models-and-libraries/dinov3-license/>

By using these weights you agree to the DINOv3 License terms, including:
- redistribution of the weights or any derivative works must be **under the
  DINOv3 License and must include a copy of it**;
- products, websites, or publications built on this model must display
  **"Built with DINOv3"** and acknowledge the use of DINOv3;
- DINOv3's prohibited-use and export-control terms apply.

Components authored by the MorphPT team (LoRA adapters, ScaleGate, classification
heads, label maps, and inference code) are released for research use consistent
with the inherited DINOv3 License above.