Vidore Benchmark 2 - World Economics report Dataset (Multilingual)

This dataset is part of the "Vidore Benchmark 2" collection, designed for evaluating visual retrieval applications. It focuses on the theme of World economic reports from 2024.

Dataset Summary

The dataset contain queries in the following languages : ["english", "french", "german", "spanish"]. Each query was originaly in "english" (see https://huggingface.co/datasets/vidore/synthetic_economics_macro_economy_2024_filtered_v1.0) and was tranlated using gpt-4o.

This dataset provides a focused benchmark for visual retrieval tasks related to World economic reports. It includes a curated set of documents, queries, relevance judgments (qrels), and page images.

• Number of Documents: 5
• Number of Queries: 232
• Number of Pages: 452
• Number of Relevance Judgments (qrels): 3628
• Average Number of Pages per Query: 15.6

Dataset Structure (Hugging Face Datasets)

The dataset is structured into the following columns:

• docs: Contains document metadata, likely including a "doc-id" field to uniquely identify each document.
• corpus: Contains page-level information:
  • "image": The image of the page (a PIL Image object).
  • "doc-id": The ID of the document this page belongs to.
  • "corpus-id": A unique identifier for this specific page within the corpus.
• queries: Contains query information:
  • "query-id": A unique identifier for the query.
  • "query": The text of the query.
  • "language": The language of the query
• qrels: Contains relevance judgments:
  • "corpus-id": The ID of the relevant page.
  • "query-id": The ID of the query.
  • "answer": Answer relevant to the query AND the page.
  • "score": The relevance score.

Usage

This dataset is designed for evaluating the performance of visual retrieval systems, particularly those focused on document image understanding.

Example Evaluation with ColPali (CLI):

Here's a code snippet demonstrating how to evaluate the ColPali model on this dataset using the vidore-benchmark command-line tool.

1. Install the vidore-benchmark package:

   pip install vidore-benchmark datasets
   

2. Run the evaluation:

   vidore-benchmark evaluate-retriever \
       --model-class colpali \
       --model-name vidore/colpali-v1.3 \
       --dataset-name vidore/economics_reports_v2 \
       --dataset-format beir \
       --split test
   

For more details on using vidore-benchmark, refer to the official documentation: https://github.com/illuin-tech/vidore-benchmark

Citation

If you use this dataset in your research or work, please cite:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}

@misc{macé2025vidorebenchmarkv2raising,
    title={ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval}, 
    author={Quentin Macé and António Loison and Manuel Faysse},
    year={2025},
    eprint={2505.17166},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
    url={https://arxiv.org/abs/2505.17166}, 
}

Acknowledgments

This work is partially supported by ILLUIN Technology, and by a grant from ANRT France.