JECT_320_aligned / README.md
lukeingawesome's picture
Initial release
0fc3616 verified
|
Raw History Blame Contribute Delete
5.53 kB
metadata
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: feature-extraction
tags:
  - medical-imaging
  - chest-ct
  - 3d
  - foundation-model
  - jepa
  - siglip

JECT_320_aligned

Model weights for JECT: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest CT.

A 3D chest-CT encoder pretrained with JECT and cooled down at 320³ on 1 mm volumes, then aligned end to end with radiology reports: the whole image encoder, a latent-attention pooler, a projection and a text adapter were trained together. A variant with the encoder kept frozen is released as JECT_320. Use it for CT embeddings, report retrieval and zero-shot finding scores.

component parameters trained in the alignment stage
image encoder (JECT, 320³ input) 161 M yes
latent-attention pooler + projection 13.7 M yes
text: chest2vec 0.6B + LoRA (r 16) + projection 11.1 M adapter adapter only

Input

A chest CT as NIfTI (.nii.gz) or as an already preprocessed cube (.npz). prepare.load_volume turns either into the model input: reorient to LPS, resample to 1 mm, pad with air (-1024 HU), crop a 384³ cube around the lungs, keep raw HU, then centre-crop 320³. The window is applied inside the model.

mode how the lungs are located speed needs
fast (default for NIfTI) air inside the body, found by HU threshold seconds monai, nibabel, scipy
totalseg TotalSegmentator lung lobes, as in training about a minute on a GPU TotalSegmentator
center middle of the scan, no lung search seconds monai, nibabel
npz cube already preprocessed for this model instant none

totalseg reproduces the training input exactly. Most chest CTs fit inside the 384³ cube, so all modes give the same input; they differ only on scans that extend beyond it (for example chest-abdomen series), where center can miss the lungs.

Measured with JECT_224 on 414 CT-RATE validation scans available as NIfTI, against the exact training input (CT-RATE-18 linear probe, macro AUROC; paired bootstrap over scans):

mode macro AUROC difference to exact [95 % CI] embedding cosine to exact, median / 5th percentile
exact (totalseg or npz) 0.875 1 / 1
fast 0.875 -0.0005 [-0.0015, +0.0005] 1.000 / 0.977
center 0.875 -0.0002 [-0.0018, +0.0014] 1.000 / 0.955
Scan type ("non-contrast", "contrast" or "ldct"): Pass the scan type; the model was trained with it.

Usage

import torch
from transformers import AutoModel

repo = "<user>/JECT_320_aligned"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda()

from huggingface_hub import snapshot_download          # input preparation and the text tower ship as code in the repo
import sys; sys.path.insert(0, snapshot_download(repo))
from prepare import load_volume
from modeling_ject import load_text_model

volume = load_volume("scan.nii.gz", model.config).cuda()        # mode="fast"; also "totalseg", "center", or an .npz path
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    out = model(volume, ct_type=["non-contrast"])
text = load_text_model(repo, device="cuda")                     # needs chest2vec/chest2vec_0.6B
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    texts = text(["[Lungs] - A 9 mm solid nodule in the right upper lobe.", "[Lungs] - The lungs are clear."])
print(model.similarity(out["embedding"].float(), texts.float()))

The model returns embedding (the 1,024-d retrieval vector), latent_embeddings (all 11 latents: 10 anatomy regions and the global one), pooled, s4_tokens (10³ = 1,000 tokens × 1,024) and s4_mean. Scores follow SigLIP: sigmoid(exp(logit_scale_log) · cos + logit_bias).

Results

CT-RATE validation, 1,564 scans and 1,503 distinct reports (identical reports count as matches), evaluated at the end of alignment training:

task R@1 R@5 R@10
image-to-report retrieval 0.330 0.580 0.668
report-to-image retrieval 0.359 0.593 0.685

Training data

Encoder: JECT pretraining at 224³ on about 173,000 chest CTs with reports (CT-RATE and a private institutional cohort), then a cooldown at 320³ on 1 mm volumes. Alignment: CT-RATE training pairs (findings section), all encoder weights trained.

Limitations

Trained on adult chest CT; reports in English. Not a medical device and not validated for clinical use. Retrieval scores reflect CT-RATE-style findings text; other report styles may score lower. The model expects the input contract above; other spacings, crops or orientations run without error but give wrong features.

Files

file content
config.json encoder architecture, readout and text settings, SigLIP scale and bias
model.safetensors image encoder, latent-attention pooler and projection
text_adapter.safetensors LoRA and projection for chest2vec 0.6B
configuration_ject.py, modeling_ject.py, ject_encoder.py, preprocess.py model code (AutoModel, trust_remote_code=True)
prepare.py NIfTI and npz input preparation

Citation

@misc{ject2026,
  title  = {{JECT}: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest {CT}},
  author = {TBD},
  year   = {2026}
}

License

Model weights: CC BY-NC-SA 4.0 (non-commercial use only; see LICENSE). Code: MIT (see LICENSE-CODE).