Expand model card

README.md

@@ -1,25 +1,91 @@
 ---
 base_model: Qwen/Qwen3-VL-8B-Instruct
+base_model_relation: adapter
 library_name: peft
 pipeline_tag: image-text-to-text
+license: apache-2.0
+language:
+- en
 tags:
 - lora
 - peft
+- cvpd
+- self-distillation
+- multimodal
+- vision-language
+- lmm
+- ocr
 - qwen3-vl
+- unsupervised
 ---
 
-# CVPD
+# CVPD: Contrastive Counterfactual Visual Process Distillation
 
-A LoRA adapter for [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct).
+This is the CVPD LoRA adapter for `Qwen/Qwen3-VL-8B-Instruct`, from our paper
+[Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots](https://github.com/mbzuai-oryx/CVPD),
+accepted to BMVC 2026.
+
+CVPD is a fully self-contained framework for dense, token-level visual self-distillation.
+Where prior visual distillation builds its privileged context from outside the model, using
+segmentation systems, region proposals, or a stronger annotator, CVPD recovers that context
+from the model's own counterfactual behavior. We train on raw, unlabeled images with no
+captions, bounding boxes, labels, reward models, or teacher models, by locating **visual blind
+spots**, regions the model can perceive but fails to exploit under full-image conditioning:
+
+- **Discovery:** the model writes its own question and probe answer per image and proposes
+  candidate regions by self-grounding and by 3x3 and 2x2 partitions. A region is kept when
+  cropping to it moves and sharpens the answer distribution while blurring ("ghosting") it
+  leaves the full-image behavior unchanged.
+- **Contrastive self-distillation:** each retained region instantiates four policies from the
+  same backbone. The online student learns from a crop-conditioned positive teacher, is pushed
+  away from a ghost-conditioned negative teacher, and is anchored to a frozen reference policy.
+
+We combine these as latent transfer plus contrastive ranking (`lambda_rank=0.5`, margin
+`m=0.1`) and optimize under a KL anchor adapted online to a target divergence of `0.03`.
 
 ## Usage
 
+This is a LoRA adapter, so load the base model first and attach the adapter:
+
 ```python
-from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
 from peft import PeftModel
 
-base = "Qwen/Qwen3-VL-8B-Instruct"
-model = Qwen3VLForConditionalGeneration.from_pretrained(base, dtype="auto", device_map="auto")
-model = PeftModel.from_pretrained(model, "shravvvv/CVPD")
-processor = AutoProcessor.from_pretrained(base)
+BASE = "Qwen/Qwen3-VL-8B-Instruct"
+ADAPTER = "shravvvv/CVPD"
+
+model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(model, ADAPTER)
+processor = AutoProcessor.from_pretrained(ADAPTER)
+model.eval()
+
+image = Image.open("example.jpg").convert("RGB")
+messages = [{"role": "user", "content": [
+    {"type": "image", "image": image},
+    {"type": "text", "text": "What is the text written on the sign?"},
+]}]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
+
+with torch.inference_mode():
+    out = model.generate(**inputs, max_new_tokens=128)
+print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
+```
+
+## License
+
+Apache 2.0.
+
+## Citation
+
+```bibtex
+@inproceedings{cvpd,
+author    = {Venkatraman, Shravan and Thawakar, Omkar and Thawkar, Ritesh and Shaker, Abdelrahman and Muhammad, Rao},
+title     = {Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots},
+booktitle = {37th British Machine Vision Conference 2026, {BMVC} 2026, Lancaster, UK, November 23-26, 2026},
+publisher = {BMVA},
+year      = {2026}
+}
 ```