README.md

---
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
  - multimodal
  - knowledge-distillation
  - compositional-reasoning
  - compodistill
---

# CompoDistill-2B

The final CompoDistill student (Qwen1.5-1.8B + SigLIP-so400m), trained with the three-stage DPT -> DFT -> SFT pipeline including visual attention distillation from the LLaVA-style Qwen1.5-4B teacher.

Released with the paper **CompoDistill: Attention Distillation for Compositional Reasoning
in Multimodal LLMs** ([arXiv:2510.12184](https://arxiv.org/abs/2510.12184)).
Training and evaluation code: https://github.com/ptkjw1997/CompoDistill

## Usage

```python
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoImageProcessor

repo = "JiwanKim/CompoDistill-2B"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True,
                                             torch_dtype=torch.float16).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(repo, use_fast=False)
image_processor = AutoImageProcessor.from_pretrained(repo)

image = Image.open("example.jpg")
print(model.chat("What is happening in this image?", tokenizer,
                 image=image, image_processor=image_processor))
```

## Citation

```bibtex
@article{kim2025compodistill,
  title={CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs},
  author={Kim, Jiwan and Kim, Kibum and Seo, Sangwoo and Park, Chanyoung},
  journal={arXiv preprint arXiv:2510.12184},
  year={2025}
}
```