--- library_name: transformers license: cc-by-nc-4.0 tags: - medical - radiology - vision-language - chest-ct - 3d-medical-imaging - glint - custom_code - arxiv:2606.03180 datasets: - ibrahimhamamci/CT-RATE --- # GLINT-CT

📝 Paper • 💻 Code • 🌐 Project page • 🤗 GLINT-CXR

GLINT-CT is the chest CT model of **GLINT** (*GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations*, NeurIPS 2026). - **Vision encoder:** V-JEPA 2.1 ViT-L/16, 384×384, 80 frames / 240 slices - **Text encoder:** `sentence-transformers/all-mpnet-base-v2` (fine-tuned) - **Alignment:** linear projection (1024→768) + LayerNorm, mean-pooled text + LayerNorm, SGA similarity head - **Training:** CT-RATE, with Sparsely Gated Alignment (SGA) and Dense Feature Regularization (DFR) from a frozen V-JEPA 2.1 ViT-G teacher > ⚠️ GLINT is released for research purposes only and is not intended for clinical use, including the diagnosis or treatment of any medical condition. ## Usage Both examples use the bundled `inference.py` / `utils.py`; run them from the downloaded repo. ### Full pipeline ```bash hf download Deepnoid/GLINT-CT --local-dir glint-ct && cd glint-ct python inference.py --volume volume.nii.gz --text "There is a lung nodule." # prints the sentence probability and saves similarity_map.nii.gz ``` `--volume` takes a NIfTI file or a DICOM series directory. The map has the input CT's shape and affine, so it overlays the input in ITK-SNAP / 3D Slicer. ### AutoModel ```python import torch from transformers import AutoModel, AutoTokenizer from utils import load_ct_volume, model_inference, save_similarity_map, similarity_scores_to_volume # bundled helpers repo = "Deepnoid/GLINT-CT" device = "cuda" if torch.cuda.is_available() else "cpu" model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).to(device).eval() tokenizer = AutoTokenizer.from_pretrained(repo) pixel_values, geometry = load_ct_volume("volume.nii.gz", return_geometry=True) # [1, 80, 3, 384, 384] prob, scores = model_inference(pixel_values, "There is a lung nodule.", model=model, tokenizer=tokenizer) similarity_map = similarity_scores_to_volume(scores, model, geometry) # input CT shape, e.g. (512, 512, 205) save_similarity_map(similarity_map, geometry["affine"], "similarity_map.nii.gz") ``` - **Outputs:** `prob` is the sentence probability (sigmoid); `scores` is the per-token similarity in [0, 1] (40 × 24 × 24 tokens); `similarity_scores_to_volume` resamples it onto the input CT grid (0 outside the 360 mm preprocessing crop), or keeps the 240×384×384 preprocessed grid without `geometry`. - **`geometry`:** returned by `load_ct_volume(..., return_geometry=True)`: `shape` and `affine` of the input CT (the map's NIfTI header) and `to_model`, the 4×4 input-voxel → preprocessed-voxel mapping used for the resampling; for array input `affine` is `None` and the map stays on the preprocessed grid. - **Inputs:** a NIfTI file or a DICOM series directory (one series); `utils.load_ct_volume` applies the training preprocessing (IPL, HU [-1000, 1000] → [-1, 1], 1.5 × 0.75 × 0.75 mm, crop/pad to 240×480×480, resize to 240×384×384) and returns `[1, 80, 3, 384, 384]`. - **Unreliable affine:** pass the voxel spacing along the reoriented I, P, L axes, e.g. `raw_spacing=(1.5, 0.7, 0.7)`, to `load_ct_volume` (`--raw_spacing` in the script). - **Underlying call:** `model.compute_logits(pixel_values, [encoded_text])` returns `logits` and `similarity_scores`. Vision encoder only (`vision_only=True`): ```python import torch from transformers import AutoModel from utils import load_ct_volume # bundled helper repo = "Deepnoid/GLINT-CT" vision = AutoModel.from_pretrained(repo, trust_remote_code=True, vision_only=True, dtype=torch.float32).eval() pixel_values = load_ct_volume("volume.nii.gz") # [1, 80, 3, 384, 384] with torch.no_grad(): out = vision(pixel_values) print(out.last_hidden_state.shape) # (1, 23040, 1024) ``` ## Requirements `transformers==4.56.1`, `torch`, `entmax`, `monai`, `nibabel`; `pydicom` for DICOM series input. > ⚠️ Keep `transformers==4.56.1`, the version the bundled code was validated with. ## Notes - Training and fine-tuning code: [GitHub](https://github.com/deepnoid-ai/GLINT); split ID lists: [`data/`](./tree/main/data). - `dfr.py` and its projector weights are used only in training; the V-JEPA 2.1 ViT-G teacher is not included. - `vision_only=True` still downloads the full checkpoint; `save_pretrained` on the loaded model keeps a vision-only copy. ## Acknowledgments This work was supported by the Technology Innovation Program (RS-2025-02221011, Development of Medical-Specialized Multimodal Hyperscale Generative AI Technology for Global Integration) funded by the Ministry of Trade, Industry & Energy (MOTIE, South Korea), and by the Advanced GPU Utilization Support Program funded by the Korean government (Ministry of Science and ICT) (Project No. 02-26-91-0330). ## License - **Weights and code:** [CC BY-NC 4.0](LICENSE). - **Vision encoder:** fine-tuned from V-JEPA 2.1 ViT-L (© Meta Platforms, Inc., MIT); `vjepa21_encoder.py` adapts [facebookresearch/vjepa2](https://github.com/facebookresearch/vjepa2) (MIT, `LICENSES/MIT.txt`). - **Text encoder:** fine-tuned from [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (Apache-2.0, modified; `LICENSES/Apache-2.0.txt`). ## Citation ```bibtex @inproceedings{park2026glint, title = {{GLINT}: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations}, author = {Park, Jonggwon and Lee, Seongeun and Park, Junhyun and Yun, Hannah and Kim, Hyunwoong and Jeong, Sohyun and Kang, Hyewon and Yoon, Byungmu and Choi, Kyoyun}, booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems}, year = {2026}, url = {https://arxiv.org/abs/2606.03180} } ```