JECT_224
Model weights for JECT: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest CT.
A 3D chest-CT encoder pretrained with JECT (hierarchical JEPA with masked section prediction from the paired report), plus a report-aligned readout. The encoder is frozen; only a latent-attention pooler, a projection and a text adapter were trained to align CT volumes with radiology reports. Use it for CT embeddings, report retrieval and zero-shot finding scores.
| component | parameters | trained in the alignment stage |
|---|---|---|
| image encoder (JECT, 224³ input) | 158 M | no |
| latent-attention pooler + projection | 13.7 M | yes |
| text: chest2vec 0.6B + LoRA (r 16) + projection | 11.1 M adapter | adapter only |
Input
A chest CT as NIfTI (.nii.gz) or as an already preprocessed cube (.npz). prepare.load_volume turns either into the
model input: reorient to LPS, resample to 1.5 mm, pad with air (-1024 HU), crop a 256³ cube around the lungs, keep raw HU,
then centre-crop 224³. The window is applied inside the model.
| mode | how the lungs are located | speed | needs |
|---|---|---|---|
fast (default for NIfTI) |
air inside the body, found by HU threshold | seconds | monai, nibabel, scipy |
totalseg |
TotalSegmentator lung lobes, as in training | about a minute on a GPU | TotalSegmentator |
center |
middle of the scan, no lung search | seconds | monai, nibabel |
npz |
cube already preprocessed for this model | instant | none |
totalseg reproduces the training input exactly. Most chest CTs fit inside the 256³ cube, so all modes give the same input;
they differ only on scans that extend beyond it (for example chest-abdomen series), where center can miss the lungs.
Measured with JECT_224 on 414 CT-RATE validation scans available as NIfTI, against the exact training input (CT-RATE-18 linear probe, macro AUROC; paired bootstrap over scans):
| mode | macro AUROC | difference to exact [95 % CI] | embedding cosine to exact, median / 5th percentile |
|---|---|---|---|
exact (totalseg or npz) |
0.875 | 1 / 1 | |
fast |
0.875 | -0.0005 [-0.0015, +0.0005] | 1.000 / 0.977 |
center |
0.875 | -0.0002 [-0.0018, +0.0014] | 1.000 / 0.955 |
Scan type ("non-contrast", "contrast" or "ldct"): Optional for this model (its scan-type embedding is zero). |
Usage
import torch
from transformers import AutoModel
repo = "<user>/JECT_224"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda()
from huggingface_hub import snapshot_download # input preparation and the text tower ship as code in the repo
import sys; sys.path.insert(0, snapshot_download(repo))
from prepare import load_volume
from modeling_ject import load_text_model
volume = load_volume("scan.nii.gz", model.config).cuda() # mode="fast"; also "totalseg", "center", or an .npz path
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
out = model(volume, ct_type=["non-contrast"])
text = load_text_model(repo, device="cuda") # needs chest2vec/chest2vec_0.6B
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
texts = text(["[Lungs] - A 9 mm solid nodule in the right upper lobe.", "[Lungs] - The lungs are clear."])
print(model.similarity(out["embedding"].float(), texts.float()))
The model returns embedding (the 1,024-d retrieval vector), latent_embeddings (all 11 latents: 10 anatomy regions and
the global one), pooled, s4_tokens (7³ = 343 tokens × 1,024) and s4_mean. Scores follow SigLIP:
sigmoid(exp(logit_scale_log) · cos + logit_bias).
Results
CT-RATE validation (1,564 scans, 1,503 distinct reports; identical reports count as matches):
| task | metric | value |
|---|---|---|
| image-to-report retrieval | R@1 / R@5 / R@10 | 0.274 / 0.524 / 0.620 |
| report-to-image retrieval | R@1 / R@5 / R@10 | 0.309 / 0.533 / 0.633 |
| zero-shot classification, 18 findings | macro AUROC ("X is present." / "X is not present.") | 0.764 |
linear probe on pooled, CT-RATE-18 labels |
macro AUROC | 0.874 |
linear probe on pooled, 44 report-derived labels |
macro AUROC | 0.853 |
RAD-ChestCT (external, 360 test scans, 16 mapped findings), linear probe on pooled trained with a fraction of the
training labels; 0 % applies the CT-RATE probe unchanged:
| labels | 0 % | 1 % | 10 % | 100 % |
|---|---|---|---|---|
| macro AUROC | 0.782 | 0.653 | 0.758 | 0.807 |
Training data
Encoder: JECT pretraining at 224³ on about 173,000 chest CTs with reports, from CT-RATE and a private institutional cohort. Alignment: 23,476 CT-RATE training pairs (findings section), encoder frozen.
Limitations
Trained on adult chest CT; reports in English. Not a medical device and not validated for clinical use. Retrieval and zero-shot scores reflect CT-RATE-style findings text; other report styles may score lower. The model expects the input contract above; other spacings, crops or orientations run without error but give wrong features.
Files
| file | content |
|---|---|
config.json |
encoder architecture, readout and text settings, SigLIP scale and bias |
model.safetensors |
image encoder, latent-attention pooler and projection |
text_adapter.safetensors |
LoRA and projection for chest2vec 0.6B |
configuration_ject.py, modeling_ject.py, ject_encoder.py, preprocess.py |
model code (AutoModel, trust_remote_code=True) |
prepare.py |
NIfTI and npz input preparation |
Citation
@misc{ject2026,
title = {{JECT}: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest {CT}},
author = {TBD},
year = {2026}
}
License
Model weights: CC BY-NC-SA 4.0 (non-commercial use only; see LICENSE). Code: MIT (see LICENSE-CODE).
- Downloads last month
- 6