Update Readme

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 data/MIMIC-CXR/*.json filter=lfs diff=lfs merge=lfs -text
+misc/*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -2,26 +2,57 @@
 license: cc-by-nc-4.0
 ---
 
-<!-- markdownlint-disable first-line-h1 -->
-<!-- markdownlint-disable html -->
-
 <div align="center">
-<h1>
-  RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Radiology with Zero-Shot Multi-Task Capability
-</h1>
-</div>
+    
+# RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability [NeurIPS 2025]
+
+
 
 <p align="center">
-📝 <a href="" target="_blank">Paper</a> • 🤗 <a href="https://huggingface.co/Deepnoid/RadZero" target="_blank">Hugging Face</a> • 🧩 <a href="" target="_blank">Github</a>
+📝 <a href="https://arxiv.org/abs/2504.07416" target="_blank">Paper</a> • 🤗 <a href="https://huggingface.co/Deepnoid/RadZero" target="_blank">Model</a> • 🧩 <a href="https://github.com/deepnoid-ai/RadZero" target="_blank">Codes</a>
 </p>
 
-<div align="center">
 </div>
 
+## Introduction 
+
+<p align="center">
+  <img src="misc/introduction.png" alt="Key Differences vs. Existing Methods" width="80%" />
+</p>
+<p align="center">
+  <em>Figure 1. Comparison of attention maps and the proposed VL similarity map for visualizing VL
+alignment. (a) While traditional attention maps inevitably exhibit high values at certain points due to softmax activation, the proposed VL similarity maps yield low values for unrelated image-text pair. (b) Their fixed scale, originating from cosine similarity, enables open-vocabulary semantic segmentation through simple thresholding.</em>
+</p>
+
+<p align="center">
+  <img src="misc/method.png" alt="RadZero Method Overview" width="80%" />
+
+</p>
+<p align="center">
+  <em>Figure 2. Overview of the RadZero framework. Finding-sentences are extracted from reports and aligned with local image patch features through similarity-based cross-attention (VL-CABS), enabling zero-shot classification, grounding, and segmentation.</em>
+</p>
+
+
+## Abstract
 
-## 🎬 Get Started
+> Recent advancements in multimodal models have significantly improved vision-language (VL) alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning and offer limited interpretability through attention probability visualizations. To address these challenges, we introduce RadZero, a novel framework for VL alignment in chest X-ray with zero-shot multi-task capability. A key component of our approach is VL-CABS (Vision-Language Cross-Attention Based on Similarity), which aligns text embeddings with local image features for interpretable, fine-grained VL reasoning. RadZero leverages large language models to extract concise semantic sentences from radiology reports and employs multi-positive contrastive training to effectively capture relationships between images and multiple relevant textual descriptions. It uses a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, VL-CABS enables zero-shot inference with similarity probability for classification, and pixel-level VL similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, VL similarity map analysis highlights the potential of VL-CABS for improving explainability in VL alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging. 
 
-```python
+
+## RadZero Model Inference
+
+### Install dependencies 
+
+```
+pip install -r requirements.txt
+```
+
+### Model Inference Codes
+
+RadZero can perform **zero-shot classification / grounding / segmentation for chest X-ray** 
+using the **RadZero model** on 🤗 <a href="https://huggingface.co/Deepnoid/RadZero" target="_blank">Hugging Face</a>.
+
+
+```
 # Deepnoid/RadZero/inference.py
 import warnings
 
@@ -76,3 +107,21 @@ if __name__ == "__main__":
     print(similarity_map.shape)
 ```
 
+
+
+
+## References
+
+- **Pretrained models**
+  - **Vision encoder**: [XrayDINOv2](https://huggingface.co/StanfordAIMI/dinov2-base-xray-224)
+  - **Text encoder**: [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
+
+
+## Acknowledgments
+This work was supported by the Technology Innovation Program (RS-2025-02221011, Development
+of Medical-Specialized Multimodal Hyperscale Generative AI Technology for Global Integration)
+funded by the Ministry of Trade Industry & Energy (MOTIE, South Korea).
+
+## LICENSE
+[![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)
+

requirements.txt

@@ -0,0 +1,265 @@
+accelerate==0.27.2
+aiofiles==23.2.1
+aiohttp==3.9.3
+aiosignal==1.3.1
+albucore==0.0.23
+albumentations==2.0.3
+altair==5.2.0
+annotated-types==0.6.0
+antlr4-python3-runtime==4.9.3
+anyio==4.2.0
+appdirs==1.4.4
+argon2-cffi==21.3.0
+argon2-cffi-bindings==21.2.0
+asttokens==2.0.5
+async-lru==2.0.4
+attrs==23.1.0
+Babel==2.11.0
+backoff==2.2.1
+beautifulsoup4==4.12.2
+bert-score==0.3.13
+bitsandbytes==0.43.0
+black==24.2.0
+bleach==4.1.0
+blessed==1.20.0
+Brotli==1.0.9
+certifi==2024.2.2
+cffi==1.16.0
+cfgv==3.4.0
+chardet==5.2.0
+charset-normalizer==2.0.4
+click==8.1.7
+colorama==0.4.6
+comm==0.2.1
+contourpy==1.2.0
+cycler==0.12.1
+datasets==2.18.0
+debugpy==1.6.7
+decorator==5.1.1
+deepspeed==0.14.0
+defusedxml==0.7.1
+dill==0.3.8
+distlib==0.3.8
+distro==1.9.0
+docker-pycreds==0.4.0
+docstring-parser==0.15
+efficientnet-pytorch==0.7.1
+einops==0.7.0
+executing==0.8.3
+f1chexbert==0.0.2
+fairscale==0.4.13
+faiss-gpu-cu12==1.8.0.2
+fastapi==0.110.0
+fastjsonschema==2.16.2
+ffmpy==0.3.2
+filelock==3.13.1
+flake8==7.0.0
+fonttools==4.49.0
+frozenlist==1.4.1
+fsspec==2024.2.0
+ftfy==6.2.0
+gitdb==4.0.11
+GitPython==3.1.42
+gpustat==1.1.1
+gradio==4.21.0
+gradio_client==0.12.0
+h11==0.14.0
+hjson==3.1.0
+httpcore==1.0.4
+httpx==0.27.0
+huggingface-hub==0.28.1
+identify==2.5.35
+idna==3.4
+imageio==2.34.1
+importlib_resources==6.1.3
+ipdb==0.13.13
+ipykernel==6.28.0
+ipython==8.20.0
+ipywidgets==8.1.2
+isort==5.13.2
+jedi==0.18.1
+Jinja2==3.1.3
+joblib==1.3.2
+json5==0.9.6
+jsonschema==4.19.2
+jsonschema-specifications==2023.7.1
+jupyter==1.0.0
+jupyter_client==8.6.0
+jupyter-console==6.6.3
+jupyter_core==5.5.0
+jupyter-events==0.8.0
+jupyter-lsp==2.2.0
+jupyter_server==2.10.0
+jupyter_server_terminals==0.4.4
+jupyterlab==4.0.11
+jupyterlab-pygments==0.1.2
+jupyterlab_server==2.25.1
+jupyterlab-widgets==3.0.10
+kaggle==1.7.4.5
+kiwisolver==1.4.5
+lazy_loader==0.4
+lightning-utilities==0.10.1
+markdown-it-py==3.0.0
+MarkupSafe==2.1.3
+matplotlib==3.8.3
+matplotlib-inline==0.1.6
+mccabe==0.7.0
+mdurl==0.1.2
+mistune==2.0.4
+mpmath==1.3.0
+msgpack==1.1.0
+multidict==6.0.5
+multiprocess==0.70.16
+munch==4.0.0
+mypy-extensions==1.0.0
+natsort==8.4.0
+nbclient==0.8.0
+nbconvert==7.10.0
+nbformat==5.9.2
+nest-asyncio==1.6.0
+networkx==3.2.1
+nibabel==5.3.2
+ninja==1.11.1.1
+nltk==3.8.1
+nodeenv==1.8.0
+notebook==7.0.8
+notebook_shim==0.2.3
+numpy==1.26.4
+nvidia-cublas-cu12==12.1.3.1
+nvidia-cuda-cupti-cu12==12.1.105
+nvidia-cuda-nvrtc-cu12==12.1.105
+nvidia-cuda-runtime-cu12==12.1.105
+nvidia-cudnn-cu12==8.9.2.26
+nvidia-cufft-cu12==11.0.2.54
+nvidia-curand-cu12==10.3.2.106
+nvidia-cusolver-cu12==11.4.5.107
+nvidia-cusparse-cu12==12.1.0.106
+nvidia-ml-py==12.535.133
+nvidia-nccl-cu12==2.19.3
+nvidia-nvjitlink-cu12==12.4.99
+nvidia-nvtx-cu12==12.1.105
+omegaconf==2.3.0
+open-clip-torch==2.24.0
+openai==1.35.7
+opencv-python==4.9.0.80
+opencv-python-headless==4.11.0.86
+orjson==3.9.15
+overrides==7.4.0
+packaging==23.1
+pandas==2.2.1
+pandocfilters==1.5.0
+parso==0.8.3
+pathspec==0.12.1
+patsy==0.5.6
+peft==0.9.0
+pexpect==4.8.0
+pillow==10.2.0
+pip==23.3.1
+pip-chill==1.0.3
+platformdirs==3.10.0
+ply==3.11
+pre-commit==3.6.2
+pretrainedmodels==0.7.4
+prometheus-client==0.14.1
+prompt-toolkit==3.0.43
+protobuf==4.25.3
+psutil==5.9.0
+ptyprocess==0.7.0
+pure-eval==0.2.2
+py-cpuinfo==9.0.0
+pyarrow==15.0.1
+pyarrow-hotfix==0.6
+pycocoevalcap==1.2
+pycocotools==2.0.8
+pycodestyle==2.11.1
+pycparser==2.21
+pydantic==2.10.6
+pydantic_core==2.27.2
+pydicom==2.4.4
+pydub==0.25.1
+pyflakes==3.2.0
+Pygments==2.15.1
+pynvml==11.5.0
+pyparsing==3.1.2
+PyQt5==5.15.10
+PyQt5-sip==12.13.0
+PySocks==1.7.1
+python-dateutil==2.8.2
+python-dotenv==1.0.1
+python-json-logger==2.0.7
+python-multipart==0.0.9
+python-slugify==8.0.4
+pytorch-lightning==2.5.0.post0
+pytz==2023.3.post1
+PyYAML==6.0.1
+pyzmq==25.1.2
+qtconsole==5.5.1
+QtPy==2.4.1
+referencing==0.30.2
+regex==2023.12.25
+requests==2.31.0
+rfc3339-validator==0.1.4
+rfc3986-validator==0.1.1
+rich==13.7.1
+rpds-py==0.10.6
+ruff==0.3.2
+safetensors==0.4.2
+scikit-image==0.23.2
+scikit-learn==1.4.1.post1
+scipy==1.12.0
+seaborn==0.13.2
+segmentation_models_pytorch==0.4.0
+semantic-version==2.10.0
+Send2Trash==1.8.2
+sentencepiece==0.2.0
+sentry-sdk==1.41.0
+setproctitle==1.3.3
+setuptools==68.2.2
+shellingham==1.5.4
+shtab==1.7.1
+simsimd==6.2.1
+sip==6.7.12
+six==1.16.0
+smmap==5.0.1
+sniffio==1.3.0
+soupsieve==2.5
+stack-data==0.2.0
+starlette==0.36.3
+statsmodels==0.14.1
+stringzilla==3.11.3
+sympy==1.12
+terminado==0.17.1
+text-unidecode==1.3
+threadpoolctl==3.3.0
+tifffile==2024.4.18
+timm==0.9.16
+tinycss2==1.2.1
+tokenizers==0.15.2
+tomlkit==0.12.0
+toolz==0.12.1
+torch==2.2.1
+torchaudio==2.2.1
+torchmetrics==1.6.1
+torchvision==0.17.1
+tornado==6.3.3
+tqdm==4.66.2
+traitlets==5.7.1
+transformers==4.39.3
+triton==2.2.0
+trl==0.7.11
+typer==0.9.0
+typing_extensions==4.12.2
+tyro==0.7.3
+tzdata==2024.1
+urllib3==2.1.0
+uvicorn==0.28.0
+virtualenv==20.25.1
+wandb==0.16.4
+wcwidth==0.2.13
+webencodings==0.5.1
+websocket-client==0.58.0
+websockets==11.0.3
+wheel==0.41.2
+widgetsnbextension==4.0.10
+xxhash==3.4.1
+yarl==1.9.4