Add CVPD model card

README.md

@@ -0,0 +1,137 @@
+---
+base_model: Qwen/Qwen3-VL-4B-Instruct
+base_model_relation: adapter
+library_name: peft
+pipeline_tag: image-text-to-text
+license: apache-2.0
+language:
+- en
+tags:
+- lora
+- peft
+- cvpd
+- self-distillation
+- self-evolving
+- multimodal
+- vision-language
+- lmm
+- ocr
+- visual-perception
+- qwen3-vl
+- unsupervised
+---
+
+# CVPD: Contrastive Counterfactual Visual Process Distillation
+
+This is the CVPD LoRA adapter for `Qwen/Qwen3-VL-4B-Instruct`, from our paper
+[Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots](https://github.com/mbzuai-oryx/CVPD).
+
+CVPD is a fully self-contained framework for dense, on-policy, token-level visual
+self-distillation. Prior visual distillation methods build their privileged context
+with external tools — segmentation models, region proposal networks, annotation
+pipelines, or stronger annotators such as GPT-4o. CVPD removes that dependency
+entirely: the privileged context is recovered from the model's *own* counterfactual
+behavior over unlabeled images, with no captions, bounding boxes, labels, reward
+models, verifiers, or teacher models at any stage.
+
+The key insight is that a region carries useful privileged information when zooming
+into it *changes and sharpens* the model's answer distribution, while erasing it
+leaves the full-image behavior largely unchanged. Such regions are **visual blind
+spots**: the model can already perceive them, but fails to exploit them under
+full-image conditioning.
+
+- **Phase 1 — Counterfactual Blind-Spot Discovery.** The model generates its own
+  fine-grained question and probe answer per image, proposes candidate regions from
+  three tracks (self-grounding, a 3x3 grid, and a 2x2 grid), and keeps only regions
+  passing a three-gate Counterfactual Criterion: latent capability divergence
+  (the crop moves the answer distribution), default perceptual invariance (the
+  ghosted image does not), and epistemic sharpening (the crop lowers entropy).
+- **Phase 2 — Contrastive Self-Distillation.** Each retained region instantiates
+  four policies from the same backbone: an online student, a crop-conditioned
+  positive teacher, a ghost-conditioned negative teacher, and a frozen reference
+  policy. The student is pulled toward the crop teacher (latent transfer), pushed
+  away from the ghost teacher (contrastive ranking, margin `m=0.1`), and anchored
+  to the reference policy (adaptive KL, target `0.03`).
+
+## Usage
+
+This is a LoRA adapter, so load the base model first and attach the adapter:
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE = "Qwen/Qwen3-VL-4B-Instruct"
+ADAPTER = "shravvvv/CVPD"
+
+model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(model, ADAPTER)
+processor = AutoProcessor.from_pretrained(ADAPTER)
+model.eval()
+
+image = Image.open("example.jpg").convert("RGB")
+messages = [{"role": "user", "content": [
+    {"type": "image", "image": image},
+    {"type": "text", "text": "What is the text written on the sign?"},
+]}]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
+
+with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=128)
+print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
+```
+
+## Results (Qwen3-VL-4B-Instruct)
+
+| Benchmark | Base | CVPD |
+| --- | --- | --- |
+| OCRBench | 81.70 | 84.35 |
+| MMStar Fine-Grained Perception | 60.86 | 63.35 |
+| MMStar Logical Reasoning | 62.96 | 65.25 |
+| MMStar Instance Reasoning | 69.83 | 71.20 |
+| InfoVQA | 77.73 | 79.15 |
+| ScienceQA | 87.51 | 89.05 |
+| AI2D | 80.10 | 82.35 |
+| CV-Bench | 85.45 | 87.15 |
+| RealWorldQA | 71.24 | 73.45 |
+| MMBench-EN | 83.51 | 84.60 |
+| MME-Perception | 1702.9 | 1715.5 |
+| SEED-Image | 78.05 | 78.20 |
+
+The largest gains appear on benchmarks requiring precise localized attention —
+OCRBench (+2.65), MMStar Fine-Grained Perception (+2.49), and MMStar Logical
+Reasoning (+2.29). CVPD is the only method in our comparison that improves over
+the base model on every benchmark without regressing on any, at both the 4B and
+8B scales. Full results, ablations, and the 8B model are in our paper.
+
+## Training
+
+- Base: `Qwen/Qwen3-VL-4B-Instruct`, vision encoder frozen.
+- LoRA: `r=32`, `alpha=64` on the attention and MLP projections of the language model backbone.
+- Optimizer: AdamW, `lr=2e-5`, weight decay `0.01`, gradient clipping `1.0`, bfloat16.
+- Objective: latent transfer + contrastive ranking (`lambda_rank=0.5`, margin `m=0.1`)
+  + KL anchor (`beta_0=1e-3`, adapted online to target KL `kappa=0.03`).
+- Momentum teacher pair via EMA of the LoRA parameters (`alpha=0.05`).
+- Discovery thresholds: `tau_crop = tau_ghost = 0.05`. All views resized to 448x448.
+- Data: 15,000 raw, unlabeled images (10,000 natural-scene + 5,000 reasoning-domain),
+  yielding ~2,590 curated blind-spot tuples (17.2% pass rate). No captions,
+  bounding boxes, labels, answers, or reward signals are used at any stage.
+
+## License
+
+Apache 2.0.
+
+## Citation
+
+```bibtex
+@article{venkatraman2026cvpd,
+  title   = {Perception Before Supervision: Self-Contained Visual Distillation
+             from Counterfactual Blind Spots},
+  author  = {Venkatraman, Shravan and Thawakar, Omkar and Thawkar, Ritesh and
+             Shaker, Abdelrahman and Anwer, Rao Muhammad},
+  year    = {2026}
+}
+```