Update README.md

README.md

@@ -39,8 +39,32 @@ license: mit
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-[More Information Needed]
-
+```python
+from transformers import AutoModel, AutoTokenizer
+from torchvision import transforms
+from PIL import Image
+
+model = AutoModel.from_pretrained("Astaxanthin/KEEP", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("Astaxanthin/KEEP", trust_remote_code=True)
+model.eval()
+transform = transforms.Compose([
+    transforms.Resize(size=224, interpolation=transforms.InterpolationMode.BICUBIC),
+    transforms.CenterCrop(size=(224, 224)),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
+])
+
+example_image_path = './quick_start/example.tif'
+example_text = ['an H&E image of breast invasive carcinoma.', 'an H&E image of normal tissue.', 'an H&E image of lung adenocarcinoma.']
+
+img_input =  transform(Image.open(example_image_path).convert('RGB')).unsqueeze(0)
+token_input = tokenizer(example_text,max_length=256,padding='max_length',truncation=True, return_tensors='pt')
+
+img_feature = model.encode_image(img_input)
+text_feature = model.encode_text(token_input)
+
+```
+<!-- 
 ### Downstream Use [optional]
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
@@ -96,7 +120,7 @@ Use the code below to get started with the model.
 
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-[More Information Needed]
+[More Information Needed] -->
 
 ## Evaluation
 
@@ -108,9 +132,9 @@ Use the code below to get started with the model.
 
 <!-- This should link to a Dataset Card if possible. -->
 
-[More Information Needed]
+We present benchmark results for a range of representative tasks. A complete set of benchmarks can be found in the [paper](https://arxiv.org/abs/2412.18***). These results will be updated with each new iteration of KEEP. 
 
-#### Factors
+<!-- #### Factors
 
 <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
@@ -121,15 +145,45 @@ Use the code below to get started with the model.
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
 [More Information Needed]
-
+ -->
 ### Results
 
-[More Information Needed]
 
-#### Summary
+#### Zero-shot Cancer Region Segmentation (DICE) 
+| Models | PLIP[[1]](https://www.nature.com/articles/s41591-023-02504-3) | QuiltNet [[2]](https://proceedings.neurips.cc/paper_files/paper/2023/hash/775ec578876fa6812c062644964b9870-Abstract-Datasets_and_Benchmarks.html) |   MI-Zero (Pub) [[3]](https://openaccess.thecvf.com/content/CVPR2023/html/Lu_Visual_Language_Pretrained_Multiple_Instance_Zero-Shot_Transfer_for_Histopathology_Images_CVPR_2023_paper.html) | CONCH [[4]](https://www.nature.com/articles/s41591-024-02856-4) | **KEEP(Ours)**  |
+|:---------------|--------------:|---------------------:|-------------------------:|-----------------:|------------------:|
+| CAMELYON16 | 0.253 | 0.157 | 0.186 | 0.292 | **0.361** |
+| PANDA | 0.295 | 0.309 | 0.276 | 0.315 | **0.334** |
+| AGGC22 | 0.284 | 0.282 | 0.324 | 0.449 | **0.530** |
+
+#### Zero-shot Cancer Detection (AUROC)
+| Models | CHIEF[[1]](https://www.nature.com/articles/s41586-024-07894-z) |   PLIP [[2]](https://www.nature.com/articles/s41591-023-02504-3)   |   QuiltNet [[3]](https://proceedings.neurips.cc/paper_files/paper/2023/hash/775ec578876fa6812c062644964b9870-Abstract-Datasets_and_Benchmarks.html)     |   MI-Zero (Pub) [[4]](https://openaccess.thecvf.com/content/CVPR2023/html/Lu_Visual_Language_Pretrained_Multiple_Instance_Zero-Shot_Transfer_for_Histopathology_Images_CVPR_2023_paper.html) |   CONCH [[5]](https://www.nature.com/articles/s41591-024-02856-4) | KEEP(Ours) |
+|:---------------|--------------:|--------------------:|-----------------:|-----------------:|------------------:| -----------------:|
+| CPTAC-CM | 0.915 | 0.970 | 0.972 | 0.985 | **0.994** | **0.994** | 
+| CPTAC-CCRCC | 0.723 | 0.330 | 0.755 | 0.886 | 0.871 | **0.999** |
+| CPTAC-PDA | 0.825 | 0.391 | 0.464 | 0.796 | 0.920 | **0.929** |
+| CPTAC-UCEC | 0.955 | 0.945 | 0.973 | 0.979 | 0.996 | **0.998** | 
+| CPTAC-LSCC | 0.901 | 0.965 | 0.966 | 0.910 | **0.987** | 0.983 | 
+| CPTAC-HNSCC | 0.946 | 0.898 | 0.874 | 0.918 | **0.982** | 0.976 | 
+| CPTAC-LUAD | 0.891 | 0.988 | 0.991 | 0.981 | 0.999 | **1.000** |  
+
+#### Zero-shot Cancer Subtyping (BACC) 
+| Models     | PLIP [[1]](https://www.nature.com/articles/s41591-023-02504-3) |   QuiltNet [[2]](https://proceedings.neurips.cc/paper_files/paper/2023/hash/775ec578876fa6812c062644964b9870-Abstract-Datasets_and_Benchmarks.html)      |   MI-Zero (Pub) [[3]](https://openaccess.thecvf.com/content/CVPR2023/html/Lu_Visual_Language_Pretrained_Multiple_Instance_Zero-Shot_Transfer_for_Histopathology_Images_CVPR_2023_paper.html)     |   CONCH [[4]](https://www.nature.com/articles/s41591-024-02856-4) |   **KEEP(Ours)**  |
+|:---------------|--------------:|---------------------------:|-------------------------:|-----------------:|------------------:|
+| TCGA-BRCA | 0.519 | 0.500 | 0.633 | 0.727 | **0.774** |
+| TCGA-NSCLC | 0.699 | 0.667 | 0.753 | 0.901 | **0.902** |
+| TCGA-RCC | 0.735 | 0.755 | 0.908 | 0.921 | **0.926** |
+| TCGA-ESCA | 0.614 | 0.746 | 0.954 | 0.923 | **0.977** |
+| TCGA-BRAIN | 0.361 | 0.346 | 0.361 | 0.453 | **0.604** |
+| UBC-OCEAN | 0.343 | 0.469 | 0.652 | **0.674** | 0.661 |
+| CPTAC-NSCLC | 0.647 | 0.607 | 0.643 | 0.836 | **0.863** |
+| EBRAINS | 0.096 | 0.093 | 0.325 | 0.371 | **0.456** |
 
+#### Summary
 
+Validated on 18 diverse benchmarks with more than 14,000 whole slide images (WSIs), KEEP achieves state-of-the-art performance in zero-shot cancer diagnostic tasks. Notably, for cancer detection, KEEP demonstrates an average sensitivity of 89.8% at a specificity of 95.0% across 7 cancer types, significantly outperforming vision-only foundation models and highlighting its promising potential for clinical application. For cancer subtyping, KEEP achieves a median balanced accuracy of 0.456 in subtyping 30 rare brain cancers, indicating strong generalizability for diagnosing rare tumors.
 
+<!-- 
 ## Model Examination [optional]
 
 <!-- Relevant interpretability work for the model goes here -->
@@ -164,7 +218,7 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 #### Software
 
-[More Information Needed]
+[More Information Needed] -->
 
 ## Citation [optional]
 
@@ -178,7 +232,7 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
   journal={arXiv preprint arXiv:2412.13126},
   year={2024}
 }
-
+<!-- 
 **APA:**
 
 [More Information Needed]
@@ -199,4 +253,4 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Contact
 
-[More Information Needed]
+[More Information Needed] -->