GLINT-CT / README.md

Resample similarity maps onto the input CT grid

db46ad7 verified 8 days ago

6.27 kB

---
library_name: transformers
license: cc-by-nc-4.0
tags:
- medical
- radiology
- vision-language
- chest-ct
- 3d-medical-imaging
- glint
- custom_code
- arxiv:2606.03180
datasets:
- ibrahimhamamci/CT-RATE
---
# GLINT-CT

<p>
📝 <a href="https://arxiv.org/abs/2606.03180">Paper</a> •
💻 <a href="https://github.com/deepnoid-ai/GLINT">Code</a> •
🌐 <a href="https://deepnoid-ai.github.io/GLINT/">Project page</a> •
🤗 <a href="https://huggingface.co/Deepnoid/GLINT-CXR">GLINT-CXR</a>
</p>

GLINT-CT is the chest CT model of **GLINT** (*GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained
Radiology Representations*, NeurIPS 2026).

- **Vision encoder:** V-JEPA 2.1 ViT-L/16, 384×384, 80 frames / 240 slices
- **Text encoder:** `sentence-transformers/all-mpnet-base-v2` (fine-tuned)
- **Alignment:** linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head
- **Training:** CT-RATE, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen
  V-JEPA 2.1 ViT-G teacher

> ⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition.

## Usage

Both examples use the bundled `inference.py` / `utils.py`; run them from the downloaded repo.

### Full pipeline

```bash
hf download Deepnoid/GLINT-CT --local-dir glint-ct && cd glint-ct
python inference.py --volume volume.nii.gz --text "There is a lung nodule."
# prints the sentence probability and saves similarity_map.nii.gz
```

`--volume` takes a NIfTI file or a DICOM series directory. The map has the input CT's shape and affine, so it
overlays the input in ITK-SNAP / 3D Slicer.

### AutoModel

```python
import torch
from transformers import AutoModel, AutoTokenizer
from utils import load_ct_volume, model_inference, save_similarity_map, similarity_scores_to_volume   # bundled helpers
repo = "Deepnoid/GLINT-CT"
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(repo)
pixel_values, geometry = load_ct_volume("volume.nii.gz", return_geometry=True)   # [1, 80, 3, 384, 384]
prob, scores = model_inference(pixel_values, "There is a lung nodule.", model=model, tokenizer=tokenizer)
similarity_map = similarity_scores_to_volume(scores, model, geometry)           # input CT shape, e.g. (512, 512, 205)
save_similarity_map(similarity_map, geometry["affine"], "similarity_map.nii.gz")
```

- **Outputs:** `prob` is the sentence probability (sigmoid); `scores` is the per-token similarity in [0, 1]
  (40 × 24 × 24 tokens); `similarity_scores_to_volume` resamples it onto the input CT grid (0 outside the 360 mm
  preprocessing crop), or keeps the 240×384×384 preprocessed grid without `geometry`.
- **`geometry`:** returned by `load_ct_volume(..., return_geometry=True)`: `shape` and `affine` of the input CT (the
  map's NIfTI header) and `to_model`, the 4×4 input-voxel → preprocessed-voxel mapping used for the resampling; for
  array input `affine` is `None` and the map stays on the preprocessed grid.
- **Inputs:** a NIfTI file or a DICOM series directory (one series); `utils.load_ct_volume` applies the training
  preprocessing (IPL, HU [-1000, 1000] → [-1, 1], 1.5 × 0.75 × 0.75 mm, crop/pad to 240×480×480, resize to
  240×384×384) and returns `[1, 80, 3, 384, 384]`.
- **Unreliable affine:** pass the voxel spacing along the reoriented I, P, L axes, e.g. `raw_spacing=(1.5, 0.7, 0.7)`,
  to `load_ct_volume` (`--raw_spacing` in the script).
- **Underlying call:** `model.compute_logits(pixel_values, [encoded_text])` returns `logits` and `similarity_scores`.

Vision encoder only (`vision_only=True`):

```python
import torch
from transformers import AutoModel
from utils import load_ct_volume   # bundled helper
repo = "Deepnoid/GLINT-CT"
vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval()
pixel_values = load_ct_volume("volume.nii.gz")   # [1, 80, 3, 384, 384]
with torch.no_grad():
    out = vision(pixel_values)
print(out.last_hidden_state.shape)               # (1, 23040, 1024)
```

## Requirements

`transformers==4.56.1`, `torch`, `entmax`, `monai`, `nibabel`; `pydicom` for DICOM series input.

> ⚠️ Keep `transformers==4.56.1`, the version the bundled code was validated with.

## Notes

- Training and fine-tuning code: [GitHub](https://github.com/deepnoid-ai/GLINT); split ID lists: [`data/`](./tree/main/data).
- `dfr.py` and its projector weights are used only in training; the V-JEPA 2.1 ViT-G teacher is not included.
- `vision_only=True` still downloads the full checkpoint; `save_pretrained` on the loaded model keeps a vision-only copy.

## Acknowledgments

This work was supported by the Technology Innovation Program (RS-2025-02221011,
Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology
for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE,
South Korea), and by the Advanced GPU Utilization Support Program funded by the
Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330).

## License

- **Weights and code:** [CC BY-NC 4.0](LICENSE).
- **Vision encoder:** fine-tuned from V-JEPA 2.1 ViT-L (© Meta Platforms, Inc., MIT); `vjepa21_encoder.py` adapts
  [facebookresearch/vjepa2](https://github.com/facebookresearch/vjepa2) (MIT, `LICENSES/MIT.txt`).
- **Text encoder:** fine-tuned from
  [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
  (Apache-2.0, modified; `LICENSES/Apache-2.0.txt`).

## Citation

```bibtex
@inproceedings{park2026glint,
  title     = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
  author    = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
               Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
               Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.03180}
}
```

	---
	library_name: transformers
	license: cc-by-nc-4.0
	tags:
	- medical
	- radiology
	- vision-language
	- chest-ct
	- 3d-medical-imaging
	- glint
	- custom_code
	- arxiv:2606.03180
	datasets:
	- ibrahimhamamci/CT-RATE
	---

	# GLINT-CT

	<p>
	📝 <a href="https://arxiv.org/abs/2606.03180">Paper</a> •
	💻 <a href="https://github.com/deepnoid-ai/GLINT">Code</a> •
	🌐 <a href="https://deepnoid-ai.github.io/GLINT/">Project page</a> •
	🤗 <a href="https://huggingface.co/Deepnoid/GLINT-CXR">GLINT-CXR</a>
	</p>

	GLINT-CT is the chest CT model of GLINT (*GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained
	Radiology Representations*, NeurIPS 2026).

	- Vision encoder: V-JEPA 2.1 ViT-L/16, 384×384, 80 frames / 240 slices
	- Text encoder: `sentence-transformers/all-mpnet-base-v2` (fine-tuned)
	- Alignment: linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head
	- Training: CT-RATE, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen
	V-JEPA 2.1 ViT-G teacher

	> ⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition.

	## Usage

	Both examples use the bundled `inference.py` / `utils.py`; run them from the downloaded repo.

	### Full pipeline

	```bash
	hf download Deepnoid/GLINT-CT --local-dir glint-ct && cd glint-ct
	python inference.py --volume volume.nii.gz --text "There is a lung nodule."
	# prints the sentence probability and saves similarity_map.nii.gz
	```

	`--volume` takes a NIfTI file or a DICOM series directory. The map has the input CT's shape and affine, so it
	overlays the input in ITK-SNAP / 3D Slicer.

	### AutoModel

	```python
	import torch
	from transformers import AutoModel, AutoTokenizer
	from utils import load_ct_volume, model_inference, save_similarity_map, similarity_scores_to_volume # bundled helpers

	repo = "Deepnoid/GLINT-CT"
	device = "cuda" if torch.cuda.is_available() else "cpu"

	model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval()
	tokenizer = AutoTokenizer.from_pretrained(repo)

	pixel_values, geometry = load_ct_volume("volume.nii.gz", return_geometry=True) # [1, 80, 3, 384, 384]
	prob, scores = model_inference(pixel_values, "There is a lung nodule.", model=model, tokenizer=tokenizer)
	similarity_map = similarity_scores_to_volume(scores, model, geometry) # input CT shape, e.g. (512, 512, 205)
	save_similarity_map(similarity_map, geometry["affine"], "similarity_map.nii.gz")
	```

	- Outputs: `prob` is the sentence probability (sigmoid); `scores` is the per-token similarity in [0, 1]
	(40 × 24 × 24 tokens); `similarity_scores_to_volume` resamples it onto the input CT grid (0 outside the 360 mm
	preprocessing crop), or keeps the 240×384×384 preprocessed grid without `geometry`.
	- `geometry`: returned by `load_ct_volume(..., return_geometry=True)`: `shape` and `affine` of the input CT (the
	map's NIfTI header) and `to_model`, the 4×4 input-voxel → preprocessed-voxel mapping used for the resampling; for
	array input `affine` is `None` and the map stays on the preprocessed grid.
	- Inputs: a NIfTI file or a DICOM series directory (one series); `utils.load_ct_volume` applies the training
	preprocessing (IPL, HU [-1000, 1000] → [-1, 1], 1.5 × 0.75 × 0.75 mm, crop/pad to 240×480×480, resize to
	240×384×384) and returns `[1, 80, 3, 384, 384]`.
	- Unreliable affine: pass the voxel spacing along the reoriented I, P, L axes, e.g. `raw_spacing=(1.5, 0.7, 0.7)`,
	to `load_ct_volume` (`--raw_spacing` in the script).
	- Underlying call: `model.compute_logits(pixel_values, [encoded_text])` returns `logits` and `similarity_scores`.

	Vision encoder only (`vision_only=True`):

	```python
	import torch
	from transformers import AutoModel
	from utils import load_ct_volume # bundled helper

	repo = "Deepnoid/GLINT-CT"
	vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval()

	pixel_values = load_ct_volume("volume.nii.gz") # [1, 80, 3, 384, 384]
	with torch.no_grad():
	out = vision(pixel_values)
	print(out.last_hidden_state.shape) # (1, 23040, 1024)
	```

	## Requirements

	`transformers==4.56.1`, `torch`, `entmax`, `monai`, `nibabel`; `pydicom` for DICOM series input.

	> ⚠️ Keep `transformers==4.56.1`, the version the bundled code was validated with.

	## Notes

	- Training and fine-tuning code: [GitHub](https://github.com/deepnoid-ai/GLINT); split ID lists: [`data/`](./tree/main/data).
	- `dfr.py` and its projector weights are used only in training; the V-JEPA 2.1 ViT-G teacher is not included.
	- `vision_only=True` still downloads the full checkpoint; `save_pretrained` on the loaded model keeps a vision-only copy.

	## Acknowledgments

	This work was supported by the Technology Innovation Program (RS-2025-02221011,
	Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology
	for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE,
	South Korea), and by the Advanced GPU Utilization Support Program funded by the
	Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330).

	## License

	- Weights and code: [CC BY-NC 4.0](LICENSE).
	- Vision encoder: fine-tuned from V-JEPA 2.1 ViT-L (© Meta Platforms, Inc., MIT); `vjepa21_encoder.py` adapts
	[facebookresearch/vjepa2](https://github.com/facebookresearch/vjepa2) (MIT, `LICENSES/MIT.txt`).
	- Text encoder: fine-tuned from
	[sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
	(Apache-2.0, modified; `LICENSES/Apache-2.0.txt`).

	## Citation

	```bibtex
	@inproceedings{park2026glint,
	title = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations},
	author = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and
	Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and
	Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun},
	booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
	year = {2026},
	url = {https://arxiv.org/abs/2606.03180}
	}
	```