Create README.md

README.md

@@ -0,0 +1,64 @@
+---
+license: mit
+language:
+- en
+tags:
+- vision-language-model
+- medical-imaging
+- computed-tomography
+- kidney-cancer
+- pytorch
+---
+
+# RenalCLIP: A Disease-Centric Vision-Language Foundation Model for Kidney Cancer
+
+RenalCLIP is a 3D Vision-Language Model (VLM) for computed tomography that leverages a novel, two-stage knowledge-enhancement pre-training strategy to excel at the comprehensive assessment of renal masses. This repository provides the official pre-trained model weights for the image and text encoders used in our study.
+
+For the full implementation, usage examples, and downstream task evaluation scripts, please visit our official GitHub repository.
+
+**GitHub:** [https://github.com/dt-yuhui/RenalCLIP](https://github.com/dt-yuhui/RenalCLIP)
+
+**Paper (arXiv):** [https://arxiv.org/abs/2508.16569](https://arxiv.org/abs/2508.16569)
+
+## Model Description
+
+The RenalCLIP model consists of two main components: a specialized image encoder and a text encoder.
+
+### Image Encoder
+
+The image encoder is designed to process 3D kidney CT volumes. Its architecture consists of a 3D ResNet-18 backbone followed by a projection layer to create embeddings suitable for cross-modal alignment.
+
+* **Checkpoint:** `RenalCLIP-image-encoder-model-best-acc.pt`
+
+### Text Encoder (LLM2Vec)
+
+The text encoder is built upon the **[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)** model and adapted into a powerful medical language expert using the **[LLM2Vec](https://github.com/McGill-NLP/llm2vec)** methodology. We provide two sets of LoRA weights corresponding to the two final stages of its pre-training:
+
+1.  **MNTP (Masked Next Token Prediction) Stage:**
+    * **LoRA Weights:** `LLM2Vec/Meta-Llama-3-8B-Instruct-radiology-ext-long/`
+    * This stage fine-tunes the base model on a large corpus of medical text (MIMIC-CXR) to enhance its general medical domain comprehension.
+
+2.  **SimCSE (Contrastive Learning) Stage:**
+    * **LoRA Weights:** `LLM2Vec/Meta-Llama-3-8B-Instruct-radiology-simcse/`
+    * This stage further refines the text encoder's understanding of kidney cancer-related terminology using our in-house pre-training corpus.
+
+**Important:** The provided text encoder checkpoints are **LoRA weights only**. To use them, you must download the base **[Llama-3-8B-Instruct model](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)** and place its files within the same directory as these LoRA weights. You may change `base_model_name_or_path` in `adapter_config.json` to load Llama correctly.
+
+## How to Use
+
+For detailed instructions on how to load the model weights, prepare data, and run the pre-training, fine-tuning, and inference scripts, please refer to our official GitHub repository:
+
+[https://github.com/dt-yuhui/RenalCLIP](https://github.com/dt-yuhui/RenalCLIP)
+
+## Citation
+
+If you find our work useful in your research, please consider citing our paper:
+
+```bibtex
+@article{Tao2025RenalCLIP,
+  title={A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer},
+  author={Yuhui Tao and Zhongwei Zhao and Zilong Wang and Xufang Luo and Feng Chen and et al.},
+  journal={arXiv preprint},
+  year={2025}
+}
+```