README.md

metadata

license: cc-by-3.0
dataset_info:
  - config_name: corpus
    features:
      - name: corpus-id
        dtype: int32
      - name: image
        dtype: image
      - name: doc-id
        dtype: string
    splits:
      - name: test
        num_bytes: 56115937
        num_examples: 452
    download_size: 54007829
    dataset_size: 56115937
  - config_name: docs
    features:
      - name: doc-id
        dtype: string
      - name: summary
        dtype: string
    splits:
      - name: test
        num_bytes: 5913
        num_examples: 5
    download_size: 9860
    dataset_size: 5913
  - config_name: qrels
    features:
      - name: query-id
        dtype: int64
      - name: corpus-id
        dtype: int64
      - name: is-answerable
        dtype: string
      - name: answer
        dtype: string
      - name: score
        dtype: int64
    splits:
      - name: test
        num_bytes: 1549516
        num_examples: 3628
    download_size: 564943
    dataset_size: 1549516
  - config_name: queries
    features:
      - name: query-id
        dtype: int64
      - name: query
        dtype: string
      - name: language
        dtype: string
      - name: gpt-4o-reasoning
        dtype: string
    splits:
      - name: test
        num_bytes: 952098
        num_examples: 232
    download_size: 492567
    dataset_size: 952098
configs:
  - config_name: corpus
    data_files:
      - split: test
        path: corpus/test-*
  - config_name: docs
    data_files:
      - split: test
        path: docs/test-*
  - config_name: qrels
    data_files:
      - split: test
        path: qrels/test-*
  - config_name: queries
    data_files:
      - split: test
        path: queries/test-*
task_categories:
  - document-question-answering
  - visual-document-retrieval

Vidore Benchmark 2 - World Economics report Dataset (Multilingual)

This dataset is part of the "Vidore Benchmark 2" collection, designed for evaluating visual retrieval applications. It focuses on the theme of World economic reports from 2024.

Dataset Summary

The dataset contain queries in the following languages : ["english", "french", "german", "spanish"]. Each query was originaly in "english" (see https://huggingface.co/datasets/vidore/synthetic_economics_macro_economy_2024_filtered_v1.0) and was tranlated using gpt-4o.

This dataset provides a focused benchmark for visual retrieval tasks related to World economic reports. It includes a curated set of documents, queries, relevance judgments (qrels), and page images.

• Number of Documents: 5
• Number of Queries: 232
• Number of Pages: 452
• Number of Relevance Judgments (qrels): 3628
• Average Number of Pages per Query: 15.6

Dataset Structure (Hugging Face Datasets)

The dataset is structured into the following columns:

• docs: Contains document metadata, likely including a "doc-id" field to uniquely identify each document.
• corpus: Contains page-level information:
  • "image": The image of the page (a PIL Image object).
  • "doc-id": The ID of the document this page belongs to.
  • "corpus-id": A unique identifier for this specific page within the corpus.
• queries: Contains query information:
  • "query-id": A unique identifier for the query.
  • "query": The text of the query.
  • "language": The language of the query
• qrels: Contains relevance judgments:
  • "corpus-id": The ID of the relevant page.
  • "query-id": The ID of the query.
  • "answer": Answer relevant to the query AND the page.
  • "score": The relevance score.

Usage

This dataset is designed for evaluating the performance of visual retrieval systems, particularly those focused on document image understanding.

Example Evaluation with ColPali (CLI):

Here's a code snippet demonstrating how to evaluate the ColPali model on this dataset using the vidore-benchmark command-line tool.

1. Install the vidore-benchmark package:

   pip install vidore-benchmark datasets
   

2. Run the evaluation:

   vidore-benchmark evaluate-retriever \
       --model-class colpali \
       --model-name vidore/colpali-v1.3 \
       --dataset-name vidore/economics_reports_v2 \
       --dataset-format beir \
       --split test
   

For more details on using vidore-benchmark, refer to the official documentation: https://github.com/illuin-tech/vidore-benchmark

Citation

If you use this dataset in your research or work, please cite:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}

@misc{macé2025vidorebenchmarkv2raising,
    title={ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval}, 
    author={Quentin Macé and António Loison and Manuel Faysse},
    year={2025},
    eprint={2505.17166},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
    url={https://arxiv.org/abs/2505.17166}, 
}

Acknowledgments

This work is partially supported by ILLUIN Technology, and by a grant from ANRT France.