README.md

---
license: cc-by-3.0
dataset_info:
- config_name: corpus
  features:
  - name: corpus-id
    dtype: int32
  - name: image
    dtype: image
  - name: doc-id
    dtype: string
  splits:
  - name: test
    num_bytes: 56115937.0
    num_examples: 452
  download_size: 54007829
  dataset_size: 56115937.0
- config_name: docs
  features:
  - name: doc-id
    dtype: string
  - name: summary
    dtype: string
  splits:
  - name: test
    num_bytes: 5913
    num_examples: 5
  download_size: 9860
  dataset_size: 5913
- config_name: qrels
  features:
  - name: query-id
    dtype: int64
  - name: corpus-id
    dtype: int64
  - name: is-answerable
    dtype: string
  - name: answer
    dtype: string
  - name: score
    dtype: int64
  splits:
  - name: test
    num_bytes: 1549516
    num_examples: 3628
  download_size: 564943
  dataset_size: 1549516
- config_name: queries
  features:
  - name: query-id
    dtype: int64
  - name: query
    dtype: string
  - name: language
    dtype: string
  - name: gpt-4o-reasoning
    dtype: string
  splits:
  - name: test
    num_bytes: 952098
    num_examples: 232
  download_size: 492567
  dataset_size: 952098
configs:
- config_name: corpus
  data_files:
  - split: test
    path: corpus/test-*
- config_name: docs
  data_files:
  - split: test
    path: docs/test-*
- config_name: qrels
  data_files:
  - split: test
    path: qrels/test-*
- config_name: queries
  data_files:
  - split: test
    path: queries/test-*
task_categories:
- document-question-answering
- visual-document-retrieval
---

# Vidore Benchmark 2 - World Economics report Dataset (Multilingual)

This dataset is part of the "Vidore Benchmark 2" collection, designed for evaluating visual retrieval applications. It focuses on the theme of **World economic reports from 2024**.

## Dataset Summary

The dataset contain queries in the following languages : ["english", "french", "german", "spanish"]. Each query was originaly in "english" (see [https://huggingface.co/datasets/vidore/synthetic_economics_macro_economy_2024_filtered_v1.0](https://huggingface.co/datasets/vidore/synthetic_economics_macro_economy_2024_filtered_v1.0])) and was tranlated using gpt-4o.

This dataset provides a focused benchmark for visual retrieval tasks related to World economic reports. It includes a curated set of documents, queries, relevance judgments (qrels), and page images.

*   **Number of Documents:** 5
*   **Number of Queries:** 232
*   **Number of Pages:** 452
*   **Number of Relevance Judgments (qrels):** 3628
*   **Average Number of Pages per Query:** 15.6

## Dataset Structure (Hugging Face Datasets)
The dataset is structured into the following columns:

*   **`docs`**:  Contains document metadata, likely including a `"doc-id"` field to uniquely identify each document.
*   **`corpus`**: Contains page-level information:
    *   `"image"`:  The image of the page (a PIL Image object).
    *   `"doc-id"`:  The ID of the document this page belongs to.
    *   `"corpus-id"`: A unique identifier for this specific page within the corpus.
*   **`queries`**: Contains query information:
    *   `"query-id"`: A unique identifier for the query.
    *   `"query"`: The text of the query.
    *   `"language"`: The language of the query
*   **`qrels`**: Contains relevance judgments:
    *   `"corpus-id"`: The ID of the relevant page.
    *   `"query-id"`: The ID of the query.
    *   `"answer"`: Answer relevant to the query AND the page.
    *   `"score"`: The relevance score.


## Usage

This dataset is designed for evaluating the performance of visual retrieval systems, particularly those focused on document image understanding.

**Example Evaluation with ColPali (CLI):**

Here's a code snippet demonstrating how to evaluate the ColPali model on this dataset using the `vidore-benchmark` command-line tool.

1.  **Install the `vidore-benchmark` package:**

    ```bash
    pip install vidore-benchmark datasets
    ```

2.  **Run the evaluation:**

    ```bash
    vidore-benchmark evaluate-retriever \
        --model-class colpali \
        --model-name vidore/colpali-v1.3 \
        --dataset-name vidore/economics_reports_v2 \
        --dataset-format beir \
        --split test
    ```
    
For more details on using `vidore-benchmark`, refer to the official documentation: [https://github.com/illuin-tech/vidore-benchmark](https://github.com/illuin-tech/vidore-benchmark)

## Citation

If you use this dataset in your research or work, please cite:

```bibtex
@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}

@misc{macé2025vidorebenchmarkv2raising,
    title={ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval}, 
    author={Quentin Macé and António Loison and Manuel Faysse},
    year={2025},
    eprint={2505.17166},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
    url={https://arxiv.org/abs/2505.17166}, 
}
```

## Acknowledgments

This work is partially supported by [ILLUIN Technology](https://www.illuin.tech/), and by a grant from ANRT France.