README.md

metadata

title: ConceptFormer Visual Document Retrieval
emoji: 🔎
colorFrom: gray
colorTo: yellow
sdk: gradio
sdk_version: 6.24.0
app_file: app.py
short_description: Rank document pages for a query with ConceptFormer
python_version: '3.12'
startup_duration_timeout: 1h

ConceptFormer — visual document retrieval

Demo of hmhm1229/ConceptFormer-Qwen, the Qwen2.5-VL-7B-Instruct checkpoint from ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval (code).

The demo reproduces the paper's encode-then-search inference path:

• documents are encoded from the page image with the prompt What is shown in this image?,
• queries are encoded as Instruct: …\nQuery: … text,
• both use last-token (EOS) pooling with L2 normalisation, and are scored by cosine similarity.

The LoRA adapter is merged into the base model at startup. The latent-concept (<|lcon|>) branch is a training-time objective and is not used at retrieval time, matching scripts/evaluate.sh upstream.

Sample pages

The bundled candidate pages come from the owid_charts_en split of ConceptFormer-Eval. The charts are published by Our World in Data under CC BY 4.0.