README.md

---
license: cc-by-nc-sa-4.0
library_name: timm
pipeline_tag: image-feature-extraction
tags:
- university-of-kentucky
- medical
- radiology
- chest-ct
- vision
- lejepa
language:
- en
---
# Model Card for DALE-CT-0-L

**Authors:** [Evan W. Damron](https://huggingface.co/evn13) · Mahmut S. Gokmen · Mitchell A. Klusty · Caroline N. Leach · Emily B. Collier · V. K. Cody Bumgardner — Institute for Biomedical Informatics Center for Applied AI (IBI-CAAI), University of Kentucky

This repository hosts the backbone weights for **DALE-CT-0-L** (Depth-Aware
Latent-Euclidean Computed Tomography — Large corpus), a foundational Vision
Transformer (ViT-Large) trained **entirely self-supervised, from scratch**, on
a ~296,000-case multi-source chest-CT pool — to our knowledge the largest
chest-CT pretraining corpus reported to date. It scales the supervision-free
[DALE-CT-0](https://huggingface.co/Kentucky-Open-Science/DALE-CT-0) recipe by ~11.5× in data
with no auxiliary labels.

**This is the recommended general-purpose DALE-CT backbone**: it achieves the
best external transfer of the 2D family (RAD-ChestCT retrained-probe AUROC
0.7572), matches the anatomically supervised DALE-CT-1S-v2 in-domain without
any labels, and preserves the anatomical world model (frozen slice embeddings
linearly decode volumetric position, R² = 0.973). For maximum in-domain
CT-RATE performance, use
[DALE-CT-2S](https://huggingface.co/Kentucky-Open-Science/DALE-CT-2S).

## Quick Load (timm)

```python
import timm
model = timm.create_model("hf-hub:Kentucky-Open-Science/DALE-CT-0-L", pretrained=True)
model.eval()
```

Inputs must be Hounsfield-Unit slices preprocessed exactly as during training
(clipping + z-score; see the full example below).

## The DALE-CT Family

All numbers are our own head-to-head measurements: every model (including the
public 3D baselines in the paper) is probed under one linear-probing MIL
protocol on shared splits (CT-RATE n = 992 test scans; RAD-ChestCT n = 360).
See the paper for the full protocol and confidence intervals.

| Model | CT-RATE Macro AUROC | RAD-ChestCT AUROC (frozen / retrained probe) | Role |
|---|---|---|---|
| [DALE-CT-0-L](https://huggingface.co/Kentucky-Open-Science/DALE-CT-0-L) ⭐ | 0.8156 | 0.6281 / **0.7572** | **Recommended general-purpose backbone** — best 2D external-transfer point estimates; supervision-free at ~287k-scan scale |
| [DALE-CT-2S](https://huggingface.co/Kentucky-Open-Science/DALE-CT-2S) | **0.8247** | 0.6252 / 0.7389 | Best in-domain (CT-RATE) |
| [DALE-CT-1S-v2](https://huggingface.co/Kentucky-Open-Science/DALE-CT-1S-v2) | 0.8098 | 0.6284 / 0.7334 | Anatomical (TotalSegmentator) dense supervision only |
| [DALE-CT-0](https://huggingface.co/Kentucky-Open-Science/DALE-CT-0) | 0.8057 | 0.5946 / 0.7477 | Pure self-supervised, CT-RATE |
| [Finetuned DINOv2](https://huggingface.co/Kentucky-Open-Science/Finetuned-DINOv2-Chest-CT) | 0.7953 | 0.6252 / 0.7550 | Continual-pretraining baseline |

**Paper:** [DALE-CT: Depth-Aware 2D Slice Encoders Learn an Anatomical World Model of Chest CT](https://arxiv.org/abs/2606.07775) · **Code:** [Kentucky-Open-Science/DALE-CT](https://github.com/Kentucky-Open-Science/DALE-CT)

## Model Details
* **Model Type:** Vision Transformer (ViT-Large) for chest CT analysis.
* **Developed by:** Institute for Biomedical Informatics Center for Applied AI (IBI-CAAI), University of Kentucky
* **Base Model Architecture:** `vit_large_patch14_dinov2` (via `timm`), randomly initialized and trained from scratch with `patch_size=16`, `img_size=512`, `in_chans=1`, `dynamic_img_size=True`.
* **Input:** 1-channel grayscale CT slice (Hounsfield Units, preprocessed as below — note the normalization statistics differ from the CT-RATE-trained variants).
* **Output:** class token and patch tokens (embedding dimension 1024).
* **License:** CC BY-NC-SA 4.0 — **non-commercial use only.** The pretraining pool includes an institutional chest-CT archive in addition to public collections; the weights are released for research use.

## Training Data
* **Corpus:** a multi-source chest-CT pool of 296,429 cases across 32 collections (287,302 used for training), stored at native resolution in true-HU form. Two cohorts dominate: the National Lung Screening Trial (~130k scans) and an institutional chest-CT archive, together roughly three-quarters of the pool; the remainder comes from ~30 public collections (RSNA pulmonary embolism, STOIC, 4D-Lung, COVID cohorts, CT-RATE, among others). See the paper for details.
* **Preprocessing:** HU clipped to `[-940.8, 923.1]` (0.5/99.5 foreground percentiles fit on the full pool), mapped to `[0, 1]`, then z-score normalized (pool mean `-25.03`, std `246.87` in HU space). **These statistics differ from the CT-RATE-trained DALE-CT variants — use the values above with this model.**

## Training Procedure
* DDP, `bf16`, 16×H100 GPUs; 3 epochs over the pool (191,357 iterations, global batch 384), no auxiliary head.
* Depth-aware multi-crop: two global 256² crops from the slab center slice and eight local 144² crops drawn from a 3-slice native-resolution axial slab; local crops guided to TotalSegmentator foreground (p=0.8). No ReX guidance (the pool lacks ReX labels outside CT-RATE).
* Objective: pure LeJEPA (invariance + SIGReg, λ=0.02); no labels of any kind.

## Preprocessing Example

```python
import torch, numpy as np, timm

model = timm.create_model("hf-hub:Kentucky-Open-Science/DALE-CT-0-L", pretrained=True)
model.eval()

clip_min, clip_max, mean_hu, std_hu = -940.8, 923.1, -25.03, 246.87  # DALE-CT-0-L stats
rng = clip_max - clip_min
norm_mean, norm_std = (mean_hu - clip_min) / rng, std_hu / rng

hu_slice = np.random.uniform(-1000, 1000, size=(512, 512))  # replace with real HU data
x = torch.from_numpy(hu_slice).float().clamp(clip_min, clip_max)
x = ((x - clip_min) / rng - norm_mean) / norm_std
x = x[None, None]  # (1, 1, H, W)

with torch.no_grad():
    cls_feature = model(x)                    # (1, 1024)
    tokens = model.forward_features(x)        # (1, 1 + N_patches, 1024)
```

## Citation
If you use this model, please cite the DALE-CT paper (https://arxiv.org/abs/2606.07775).