Update README.md

README.md

@@ -5,7 +5,7 @@ license: apache-2.0
 # Model Details
 
 Perception Encoder (PE) is a state-of-the-art encoder for image and video understanding trained via simple vision-language learning. It was introduced in "[Perception Encoder: The best visual embeddings
-are hidden inside the network](https://TBC)".
+are not at the output of the network](https://ai.meta.com/research/publications/perception-encoder-the-best-visual-embeddings-are-not-at-the-output-of-the-network/)".
 
 **Model Developer**: Meta
 
@@ -16,22 +16,21 @@ are hidden inside the network](https://TBC)".
 
 | Scale | Tower | Params | Width | Depth | MLP | Heads | CLIP Dim | Resolution | Patch Size | Text Context Length |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| **B** | Vision | 0.09B | 768 | 12 | 3072 | 12 | 1024 | 384 | 16 | 32 |
-|  | Text | 0.31B | 1024 | 24 | 4096 | 16 | 1024 | 384 | 16 | 32 |
+| **B** | Vision | 0.09B | 768 | 12 | 3072 | 12 | 1024 | 224 | 16 | 32 |
+|  | Text | 0.31B | 1024 | 24 | 4096 | 16 | 1024 | 224 | 16 | 32 |
 | **L** | Vision | 0.32B | 1024 | 24 | 4096 | 16 | 1024 | 336 | 14 | 32 |
 |  | Text | 0.31B | 1024 | 24 | 4096 | 16 | 1024 | 336 | 14 | 32 |
-| **G** | Vision | 1.88B | 1536 | 50 | 8960 | 16 | 1280 | 392 | 14 | 72 |
-|  | Text | 0.47B | 1280 | 24 | 5120 | 20 | 1280 | 392 | 14 | 72 |
+| **G** | Vision | 1.88B | 1536 | 50 | 8960 | 16 | 1280 | 448 | 14 | 72 |
+|  | Text | 0.47B | 1280 | 24 | 5120 | 20 | 1280 | 448 | 14 | 72 |
 
 
 # How to use
 
 ## PE codebase
-We provide the pretraining code in https://github.com/meta-ai-research-fair/occhi.git
-
+We provide the pretraining code in https://github.com/facebookresearch/perception_models
 ```shell
-git clone https://github.com/meta-ai-research-fair/occhi.git
-cd occhi
+git clone https://github.com/facebookresearch/perception_models.git
+cd perception_models
 conda create --name occhi-env python=3.12
 conda activate occhi-env
 # Install PyTorch
@@ -46,8 +45,10 @@ pip install -e .
 import torch
 from occhi.vision_encoder.factory import create_model_and_transforms, get_tokenizer
 from PIL import Image
-model_name = 'PEv1-G-14'
-pretrained='ckpts/pev1_gs14_448_rc2.pt'
+
+model_name = 'PEv1-L14-336'
+pretrained='PATH_TO_PE_Core_L14_336'
+
 model, _, preprocess = create_model_and_transforms(
     model_name,
     pretrained=pretrained,
@@ -56,40 +57,31 @@ model = model.cuda()
 tokenizer = get_tokenizer(model_name)
 image = preprocess(Image.open("docs/cat.png")).unsqueeze(0).cuda()
 text = tokenizer(["a diagram", "a dog", "a cat"]).cuda()
+
 with torch.no_grad(), torch.autocast("cuda"):
     image_features = model.encode_image(image)
     text_features = model.encode_text(text)
     image_features /= image_features.norm(dim=-1, keepdim=True)
     text_features /= text_features.norm(dim=-1, keepdim=True)
     text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+
 print("Label probs:", text_probs)  # prints: [[0.0, 0.0, 1.0]]
 ```
-
 You can find more details in the GitHub repo.
-
-
 # Evaluation
-We evaluate the pretrained MobileLLM models on Zero-shot Common Sense Reasoning tasks
-
+We evaluate the pretrained PE models on Zero-shot Common Sense Reasoning tasks
 Here is the table in Markdown format:
-
 ## Zero-Shot Image Results
-
 <img src="https://huggingface.co/facebook/PE-Core-G14-448/resolve/main/docs/pe_zeroshot_image.png" style="width: 100%; margin: 0;" />
-
 ## Zero-Shot Video Results
-
 <img src="https://huggingface.co/facebook/PE-Core-G14-448/resolve/main/docs/pe_zeroshot_video.png" style="width: 90%; margin: 0" />
-
-
 # Citation
-
 If you find our code useful for your research, please consider citing:
     
 	@article{PE,
-	    title={Perception Encoder},
+	    title={Perception Encoder: The best visual embeddings are not at the output of the network},
 	    author={},
 	    journal={arXiv:xxx.xxxxx},
 	    year={2025}
 	}
-    
+