|
Download README.md from lukeingawesome/JECT_320_aligned: direct link, hf CLI and curl.
- Browser
- Download file 5.53 kB
-
https://huggingface.co/lukeingawesome/JECT_320_aligned/resolve/main/README.md
- Command line
-
hf download hf://lukeingawesome/JECT_320_aligned/README.md
-
curl -L -o README.md https://huggingface.co/lukeingawesome/JECT_320_aligned/resolve/main/README.md
5.53 kB
| license: cc-by-nc-sa-4.0 | |
| library_name: pytorch | |
| pipeline_tag: feature-extraction | |
| tags: | |
| - medical-imaging | |
| - chest-ct | |
| - 3d | |
| - foundation-model | |
| - jepa | |
| - siglip | |
| # JECT_320_aligned | |
| Model weights for *JECT: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest CT*. | |
| A 3D chest-CT encoder pretrained with JECT and cooled down at 320³ on 1 mm volumes, then aligned end to end with | |
| radiology reports: the whole image encoder, a latent-attention pooler, a projection and a text adapter were trained | |
| together. A variant with the encoder kept frozen is released as JECT_320. Use it for CT embeddings, report retrieval and | |
| zero-shot finding scores. | |
| | component | parameters | trained in the alignment stage | | |
| |---|---|---| | |
| | image encoder (JECT, 320³ input) | 161 M | yes | | |
| | latent-attention pooler + projection | 13.7 M | yes | | |
| | text: chest2vec 0.6B + LoRA (r 16) + projection | 11.1 M adapter | adapter only | | |
| ## Input | |
| A chest CT as NIfTI (`.nii.gz`) or as an already preprocessed cube (`.npz`). `prepare.load_volume` turns either into the | |
| model input: reorient to LPS, resample to 1 mm, pad with air (-1024 HU), crop a 384³ cube around the lungs, keep raw HU, | |
| then centre-crop 320³. The window is applied inside the model. | |
| | mode | how the lungs are located | speed | needs | | |
| |---|---|---|---| | |
| | `fast` (default for NIfTI) | air inside the body, found by HU threshold | seconds | `monai`, `nibabel`, `scipy` | | |
| | `totalseg` | TotalSegmentator lung lobes, as in training | about a minute on a GPU | `TotalSegmentator` | | |
| | `center` | middle of the scan, no lung search | seconds | `monai`, `nibabel` | | |
| | `npz` | cube already preprocessed for this model | instant | none | | |
| `totalseg` reproduces the training input exactly. Most chest CTs fit inside the 384³ cube, so all modes give the same input; | |
| they differ only on scans that extend beyond it (for example chest-abdomen series), where `center` can miss the lungs. | |
| Measured with JECT_224 on 414 CT-RATE validation scans available as NIfTI, against the exact training input (CT-RATE-18 | |
| linear probe, macro AUROC; paired bootstrap over scans): | |
| | mode | macro AUROC | difference to exact [95 % CI] | embedding cosine to exact, median / 5th percentile | | |
| |---|---|---|---| | |
| | exact (`totalseg` or `npz`) | 0.875 | | 1 / 1 | | |
| | `fast` | 0.875 | -0.0005 [-0.0015, +0.0005] | 1.000 / 0.977 | | |
| | `center` | 0.875 | -0.0002 [-0.0018, +0.0014] | 1.000 / 0.955 | | |
| Scan type (`"non-contrast"`, `"contrast"` or `"ldct"`): Pass the scan type; the model was trained with it. | |
| ## Usage | |
| ```python | |
| import torch | |
| from transformers import AutoModel | |
| repo = "<user>/JECT_320_aligned" | |
| model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda() | |
| from huggingface_hub import snapshot_download # input preparation and the text tower ship as code in the repo | |
| import sys; sys.path.insert(0, snapshot_download(repo)) | |
| from prepare import load_volume | |
| from modeling_ject import load_text_model | |
| volume = load_volume("scan.nii.gz", model.config).cuda() # mode="fast"; also "totalseg", "center", or an .npz path | |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| out = model(volume, ct_type=["non-contrast"]) | |
| text = load_text_model(repo, device="cuda") # needs chest2vec/chest2vec_0.6B | |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| texts = text(["[Lungs] - A 9 mm solid nodule in the right upper lobe.", "[Lungs] - The lungs are clear."]) | |
| print(model.similarity(out["embedding"].float(), texts.float())) | |
| ``` | |
| The model returns `embedding` (the 1,024-d retrieval vector), `latent_embeddings` (all 11 latents: 10 anatomy regions and | |
| the global one), `pooled`, `s4_tokens` (10³ = 1,000 tokens × 1,024) and `s4_mean`. Scores follow SigLIP: | |
| `sigmoid(exp(logit_scale_log) · cos + logit_bias)`. | |
| ## Results | |
| CT-RATE validation, 1,564 scans and 1,503 distinct reports (identical reports count as matches), evaluated at the end of | |
| alignment training: | |
| | task | R@1 | R@5 | R@10 | | |
| |---|---|---|---| | |
| | image-to-report retrieval | 0.330 | 0.580 | 0.668 | | |
| | report-to-image retrieval | 0.359 | 0.593 | 0.685 | | |
| ## Training data | |
| Encoder: JECT pretraining at 224³ on about 173,000 chest CTs with reports (CT-RATE and a private institutional cohort), | |
| then a cooldown at 320³ on 1 mm volumes. Alignment: CT-RATE training pairs (findings section), all encoder weights trained. | |
| ## Limitations | |
| Trained on adult chest CT; reports in English. Not a medical device and not validated for clinical use. Retrieval scores | |
| reflect CT-RATE-style findings text; other report styles may score lower. The model expects the input contract above; | |
| other spacings, crops or orientations run without error but give wrong features. | |
| ## Files | |
| | file | content | | |
| |---|---| | |
| | `config.json` | encoder architecture, readout and text settings, SigLIP scale and bias | | |
| | `model.safetensors` | image encoder, latent-attention pooler and projection | | |
| | `text_adapter.safetensors` | LoRA and projection for chest2vec 0.6B | | |
| | `configuration_ject.py`, `modeling_ject.py`, `ject_encoder.py`, `preprocess.py` | model code (`AutoModel`, `trust_remote_code=True`) | | |
| | `prepare.py` | NIfTI and npz input preparation | | |
| ## Citation | |
| ```bibtex | |
| @misc{ject2026, | |
| title = {{JECT}: Addressing the Context Shortcut in Joint-Embedding Predictive Learning for Chest {CT}}, | |
| author = {TBD}, | |
| year = {2026} | |
| } | |
| ``` | |
| ## License | |
| Model weights: CC BY-NC-SA 4.0 (non-commercial use only; see `LICENSE`). Code: MIT (see `LICENSE-CODE`). | |