Add model card

README.md

@@ -1,55 +1,94 @@
 ---
+license: mit
 library_name: transformers
+pipeline_tag: zero-shot-image-classification
+base_model: openai/clip-vit-base-patch32
+language:
+- en
 tags:
-- generated_from_trainer
-datasets:
-- coco-karpathy-with-image
-model-index:
-- name: clip-large-zero-T0.1-P0.5-h3-l1-lr1e-5-250404
-  results: []
+- clip
+- vision-language
+- compositional-reasoning
+- contrastive-learning
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# READ-CLIP (ViT-B/32)
 
-# clip-large-zero-T0.1-P0.5-h3-l1-lr1e-5-250404
+**READ-CLIP** is a CLIP model fine-tuned with **READ** (**RE**construction and **A**lignment of text **D**escriptions), a lightweight recipe that strengthens the compositional reasoning of vision–language models. This is the official checkpoint for the NeurIPS 2025 paper *"Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions."*
 
-This model is a fine-tuned version of [](https://huggingface.co/) on the coco-karpathy-with-image dataset.
+- 📄 **Paper:** [arXiv:2510.16540](https://arxiv.org/abs/2510.16540) (NeurIPS 2025)
+- 💻 **Code:** [github.com/JiH00nKw0n/READ-CLIP](https://github.com/JiH00nKw0n/READ-CLIP)
+- 🧩 **Base model:** [`openai/clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32)
 
-## Model description
+## Method
 
-More information needed
+Contrastively trained CLIP models tend to behave like a bag of words, attending to individual tokens rather than the relationships between them. READ adds two auxiliary objectives on top of the standard contrastive loss during fine-tuning:
 
-## Intended uses & limitations
+- **Token-level reconstruction** — a *frozen* T5 decoder (`google/t5-v1_1-large`) reconstructs related captions from the CLIP text embedding, forcing the embedding to retain word-relationship information.
+- **Sentence-level alignment** — paraphrases of the same caption are pulled together in the embedding space, making representations robust to surface wording.
 
-More information needed
+Both objectives are **training-only**. At inference, READ-CLIP is a drop-in `CLIPModel`: no decoder, no extra parameters, and the same compute as the original CLIP.
 
-## Training and evaluation data
+## Usage
 
-More information needed
+The checkpoint loads directly with `transformers` as a standard `CLIPModel`:
 
-## Training procedure
+```python
+import torch
+from transformers import CLIPModel, CLIPProcessor
 
-### Training hyperparameters
+device = "cuda" if torch.cuda.is_available() else "cpu"
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 256
-- eval_batch_size: 8
-- seed: 42
-- distributed_type: multi-GPU
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 50
-- num_epochs: 5
+model = CLIPModel.from_pretrained("Mayfull/READ-CLIP").to(device)
+processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
 
-### Training results
+inputs = processor(
+    text=["a photo of a cat", "a photo of a dog"],
+    images=image,                 # a PIL.Image
+    return_tensors="pt",
+    padding=True,
+).to(device)
 
+with torch.no_grad():
+    outputs = model(**inputs)
 
+probs = outputs.logits_per_image.softmax(dim=-1)
+```
 
-### Framework versions
+## Results
 
-- Transformers 4.48.3
-- Pytorch 2.6.0+cu124
-- Datasets 3.2.0
-- Tokenizers 0.21.0
+Compositional reasoning accuracy on five standard benchmarks (ViT-B/32 backbone):
+
+| Benchmark           | READ-CLIP | NegCLIP | FSC-CLIP |
+|---------------------|:---------:|:-------:|:--------:|
+| WhatsUp             | **43.9**  | 42.4    | 39.8     |
+| VALSE               | **76.2**  | 73.7    | 74.4     |
+| CREPE               | 41.5      | 30.5    | **42.5** |
+| SugarCrepe          | **87.0**  | 83.6    | 85.2     |
+| SugarCrepe++ (ITT)  | **69.8**  | 65.0    | 67.9     |
+| SugarCrepe++ (TOT)  | **66.2**  | 62.5    | 64.4     |
+| **Average**         | **64.1**  | 59.6    | 62.4     |
+
+See the [paper](https://arxiv.org/abs/2510.16540) for the full set of baselines and ablations.
+
+## Training
+
+- **Backbone:** `openai/clip-vit-base-patch32` (ViT-B/32)
+- **Data:** MS-COCO (Karpathy training split, ~113K image–caption pairs)
+- **Schedule:** 5 epochs, global batch size 256, AdamW, lr 1e-5 (cosine), weight decay 0.1, bf16
+- **Hardware:** 1× NVIDIA A100 (~2 GPU-hours), seed 2025
+
+## Citation
+
+```bibtex
+@inproceedings{kwon2025readclip,
+  title     = {Enhancing Compositional Reasoning in {CLIP} via Reconstruction and Alignment of Text Descriptions},
+  author    = {Kwon, Jihoon and Min, Kyle and Sohn, Jy-yong},
+  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
+  year      = {2025}
+}
+```
+
+## License
+
+Released under the [MIT License](https://github.com/JiH00nKw0n/READ-CLIP/blob/master/LICENSE).