README.md

---
title: ConceptFormer Visual Document Retrieval
emoji: 🔎
colorFrom: gray
colorTo: yellow
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
short_description: Rank document pages for a query with ConceptFormer
python_version: "3.12"
startup_duration_timeout: 1h
---

# ConceptFormer — visual document retrieval

Demo of [`hmhm1229/ConceptFormer-Qwen`](https://huggingface.co/hmhm1229/ConceptFormer-Qwen),
the Qwen2.5-VL-7B-Instruct checkpoint from
[*ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual
Document Retrieval*](https://huggingface.co/papers/2608.15698)
([code](https://github.com/NEUIR/ConceptFormer)).

The demo reproduces the paper's encode-then-search inference path:

* documents are encoded from the page image with the prompt `What is shown in this image?`,
* queries are encoded as `Instruct: …\nQuery: …` text,
* both use last-token (EOS) pooling with L2 normalisation, and are scored by cosine similarity.

The LoRA adapter is merged into the base model at startup. The latent-concept (`<|lcon|>`)
branch is a training-time objective and is not used at retrieval time, matching
`scripts/evaluate.sh` upstream.

## Sample pages

The bundled candidate pages come from the `owid_charts_en` split of
[ConceptFormer-Eval](https://huggingface.co/datasets/hmhm1229/ConceptFormer-Eval).
The charts are published by [Our World in Data](https://ourworldindata.org) under
CC BY 4.0.