README.md

---
license: cc-by-4.0
datasets:
- HaHaJun1101/OACIRR
base_model:
- Salesforce/blip2-itm-vit-g
- Salesforce/blip2-itm-vit-g-coco
library_name: pytorch
tags:
  - composed-image-retrieval
  - object-anchored
  - image-retrieval
  - vision-language
  - multimodal
  - cvpr2026
---

# **🔍 Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval (CVPR 2026 Highlight)**

[**📖 Paper (arXiv)**](https://arxiv.org/abs/2604.05393) | [**🌐 Homepage**](https://hahajun1101.github.io/OACIR/) | [**🐙 Code (GitHub)**](https://github.com/HaHaJun1101/OACIR) | [**🤗 Dataset (OACIRR)**](https://huggingface.co/datasets/HaHaJun1101/OACIRR) | <a href="#downloading-the-adafocal-weights" style="color: red;">**🛜 Download Weights Now 👇**</a>

---

## 🔔 News
- **🌟 [2026-04-09]: Our paper has been selected as a ✨*Highlight*✨ at CVPR 2026!**
- **🔥 [2026-04-07]: The *AdaFocal* model checkpoints are officially released and are now available for use!**
- **🔥 [2026-04-03]: The full Training/Evaluation code are officially released on GitHub!**
- **🔥 [2026-03-25]: The OACIRR Benchmark is officially released on HuggingFace!**
- **🎉 [2026-02-21]: Our paper "Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval" has been accepted to CVPR 2026!**

---

## 🤖 Model Description

- **Architecture: ViT-G (EVA-CLIP) + BLIP-2 Q-Former + Context-Aware Attention Modulator (CAAM)**
- **Task: Fine-grained Composed Image Retrieval (CIR) with Instance-level Consistency**
- **Training Data: Exclusively trained on the [OACIRR Union Dataset](https://huggingface.co/datasets/HaHaJun1101/OACIRR/tree/main)**

---

## ⚙️ AdaFocal Framework

To address the core challenges of the OACIR task, we propose **AdaFocal**, an effective framework that dynamically modulates visual attention for precise, instance-level retrieval. Our approach augments a multimodal fusion backbone with a lightweight **Context-Aware Attention Modulator (CAAM)**, enabling a nuanced balance between instance fidelity and compositional reasoning.

<p align="left">
  <img 
    src="https://huggingface.co/datasets/HaHaJun1101/OACIRR/resolve/main/figures/AdaFocal_framework.png" 
    width="100%" 
    alt="AdaFocal Framework Overview"
  />
</p>

Specifically, **AdaFocal** employs a two-stage reasoning process: *Contextual Perception* and *Adaptive Focus*. It first perceives the query's compositional context to predict a modulation scalar (β). This learned signal then drives an Attention Activation Mechanism, which explicitly and adaptively intensifies the visual focus on the user-specified instance region (provided via bounding box) during multimodal feature fusion.

By dynamically re-weighting the attention distribution, **AdaFocal** seamlessly synthesizes the anchored instance, the global visual scene, and the textual modification into a coherent representation, establishing a robust and flexible baseline for identity-preserving retrieval.

---

## 🚀 How to Use

<a name="downloading-the-adafocal-weights"></a>

### 1. Download the AdaFocal Weights

You can download the checkpoints using Git LFS:
```bash
cd OACIR
git lfs install
git clone https://huggingface.co/HaHaJun1101/AdaFocal ./checkpoints
```

Alternatively, download them via the Hugging Face Python API:
```python
from huggingface_hub import snapshot_download

snapshot_download(repo_id="HaHaJun1101/AdaFocal", local_dir="OACIR/checkpoints", repo_type="model")
```

### 2. Run Evaluation via Official Codebase

Once downloaded, you can directly evaluate the models using the `evaluate.sh` script provided in our GitHub codebase. Open `evaluate.sh` and set the path to your downloaded weights:
```bash
# Inside evaluate.sh
DATASET="Fashion"
MODEL_NAME="oacir_adafocal"
MODEL_WEIGHT="./checkpoints/adafocal_scalar.pt"  # or adafocal_vector.pt
```
Then execute the script:
```bash
bash evaluate.sh
```

---

## 🏆 Model Performance on OACIRR

We provide two variants of the **AdaFocal** weights. You can instantly reproduce the following results using our provided `evaluate.sh` script.

| Model Variant | Component Type | R<sub>ID</sub>@1 (Avg) | R@1 (Avg) | R@5 (Avg) | Overall Avg | Weights File |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
| **AdaFocal (Scalar β)** | Default Configuration | 81.52 | 63.08 | 90.98 | **78.53** | [`adafocal_scalar.pt`](https://huggingface.co/HaHaJun1101/AdaFocal/blob/main/adafocal_scalar.pt) |
| **AdaFocal (Vector β)** | Vector Ablation | 81.99 | 63.06 | 91.35 | **78.80** | [`adafocal_vector.pt`](https://huggingface.co/HaHaJun1101/AdaFocal/blob/main/adafocal_vector.pt) |

*Detailed breakdowns across the 4 domains:*

| Variant | <font color=#990000>Fashion</font> (R<sub>ID</sub>@1 / R@1) | <font color=#CC3300>Car</font> (R<sub>ID</sub>@1 / R@1) | <font color=#003399>Product</font> (R<sub>ID</sub>@1 / R@1) | <font color=#006633>Landmark</font> (R<sub>ID</sub>@1 / R@1) | 
|:---|:---:|:---:|:---:|:---:|
| **Scalar β** | 73.68 / 64.45 | 78.39 / 54.85 | 91.36 / 73.85 | 82.65 / 59.18 |
| **Vector β** | 75.71 / 65.97 | 77.97 / 54.35 | 91.39 / 73.30 | 82.90 / 58.63 |

---

## ✒️ Citation

If you find our dataset, models, or codebase useful in your research, please consider citing our paper:
```bibtex
@article{yang2026beyond,
  title={Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval},
  author={Yang, Yuxin and Zhou, Yinan and Chen, Yuxin and Zhang, Ziqi and Ma, Zongyang and Yuan, Chunfeng and Li, Bing and Gao, Jun and Hu, Weiming},
  journal={arXiv preprint arXiv:2604.05393},
  year={2026}
}
```