File size: 5,526 Bytes
0fc3616
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
---
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: feature-extraction
tags:
- medical-imaging
- chest-ct
- 3d
- foundation-model
- jepa
- siglip
---

# JECT_320_aligned

Model weights for *JECT: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest CT*.

A 3D chest-CT encoder pretrained with JECT and cooled down at 320³ on 1 mm volumes, then aligned end to end with
radiology reports: the whole image encoder, a latent-attention pooler, a projection and a text adapter were trained
together. A variant with the encoder kept frozen is released as JECT_320. Use it for CT embeddings, report retrieval and
zero-shot finding scores.

| component | parameters | trained in the alignment stage |
|---|---|---|
| image encoder (JECT, 320³ input) | 161 M | yes |
| latent-attention pooler + projection | 13.7 M | yes |
| text: chest2vec 0.6B + LoRA (r 16) + projection | 11.1 M adapter | adapter only |

## Input

A chest CT as NIfTI (`.nii.gz`) or as an already preprocessed cube (`.npz`). `prepare.load_volume` turns either into the
model input: reorient to LPS, resample to 1 mm, pad with air (-1024 HU), crop a 384³ cube around the lungs, keep raw HU,
then centre-crop 320³. The window is applied inside the model.

| mode | how the lungs are located | speed | needs |
|---|---|---|---|
| `fast` (default for NIfTI) | air inside the body, found by HU threshold | seconds | `monai`, `nibabel`, `scipy` |
| `totalseg` | TotalSegmentator lung lobes, as in training | about a minute on a GPU | `TotalSegmentator` |
| `center` | middle of the scan, no lung search | seconds | `monai`, `nibabel` |
| `npz` | cube already preprocessed for this model | instant | none |

`totalseg` reproduces the training input exactly. Most chest CTs fit inside the 384³ cube, so all modes give the same input;
they differ only on scans that extend beyond it (for example chest-abdomen series), where `center` can miss the lungs.

Measured with JECT_224 on 414 CT-RATE validation scans available as NIfTI, against the exact training input (CT-RATE-18
linear probe, macro AUROC; paired bootstrap over scans):

| mode | macro AUROC | difference to exact [95 % CI] | embedding cosine to exact, median / 5th percentile |
|---|---|---|---|
| exact (`totalseg` or `npz`) | 0.875 | | 1 / 1 |
| `fast` | 0.875 | -0.0005 [-0.0015, +0.0005] | 1.000 / 0.977 |
| `center` | 0.875 | -0.0002 [-0.0018, +0.0014] | 1.000 / 0.955 |
Scan type (`"non-contrast"`, `"contrast"` or `"ldct"`): Pass the scan type; the model was trained with it.

## Usage

```python
import torch
from transformers import AutoModel

repo = "<user>/JECT_320_aligned"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda()

from huggingface_hub import snapshot_download          # input preparation and the text tower ship as code in the repo
import sys; sys.path.insert(0, snapshot_download(repo))
from prepare import load_volume
from modeling_ject import load_text_model

volume = load_volume("scan.nii.gz", model.config).cuda()        # mode="fast"; also "totalseg", "center", or an .npz path
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    out = model(volume, ct_type=["non-contrast"])
text = load_text_model(repo, device="cuda")                     # needs chest2vec/chest2vec_0.6B
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    texts = text(["[Lungs] - A 9 mm solid nodule in the right upper lobe.", "[Lungs] - The lungs are clear."])
print(model.similarity(out["embedding"].float(), texts.float()))
```

The model returns `embedding` (the 1,024-d retrieval vector), `latent_embeddings` (all 11 latents: 10 anatomy regions and
the global one), `pooled`, `s4_tokens` (10³ = 1,000 tokens × 1,024) and `s4_mean`. Scores follow SigLIP:
`sigmoid(exp(logit_scale_log) · cos + logit_bias)`.

## Results

CT-RATE validation, 1,564 scans and 1,503 distinct reports (identical reports count as matches), evaluated at the end of
alignment training:

| task | R@1 | R@5 | R@10 |
|---|---|---|---|
| image-to-report retrieval | 0.330 | 0.580 | 0.668 |
| report-to-image retrieval | 0.359 | 0.593 | 0.685 |

## Training data

Encoder: JECT pretraining at 224³ on about 173,000 chest CTs with reports (CT-RATE and a private institutional cohort),
then a cooldown at 320³ on 1 mm volumes. Alignment: CT-RATE training pairs (findings section), all encoder weights trained.

## Limitations

Trained on adult chest CT; reports in English. Not a medical device and not validated for clinical use. Retrieval scores
reflect CT-RATE-style findings text; other report styles may score lower. The model expects the input contract above;
other spacings, crops or orientations run without error but give wrong features.

## Files

| file | content |
|---|---|
| `config.json` | encoder architecture, readout and text settings, SigLIP scale and bias |
| `model.safetensors` | image encoder, latent-attention pooler and projection |
| `text_adapter.safetensors` | LoRA and projection for chest2vec 0.6B |
| `configuration_ject.py`, `modeling_ject.py`, `ject_encoder.py`, `preprocess.py` | model code (`AutoModel`, `trust_remote_code=True`) |
| `prepare.py` | NIfTI and npz input preparation |

## Citation

```bibtex
@misc{ject2026,
  title  = {{JECT}: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest {CT}},
  author = {TBD},
  year   = {2026}
}
```

## License

Model weights: CC BY-NC-SA 4.0 (non-commercial use only; see `LICENSE`). Code: MIT (see `LICENSE-CODE`).