| | --- |
| | title: DISBench Leaderboard |
| | emoji: 🏆 |
| | colorFrom: yellow |
| | colorTo: red |
| | sdk: docker |
| | app_port: 7860 |
| | pinned: true |
| | --- |
| | |
| | # 🏆 DISBench Leaderboard |
| |
|
| | Welcome to the official leaderboard for **DISBench (DeepImageSearch Benchmark)**! |
| |
|
| | DISBench is a comprehensive benchmark for evaluating DeepImageSearch methods on photo collections. This leaderboard tracks and compares the performance of various approaches on standardized evaluation metrics. |
| |
|
| | ## 📊 Evaluation Metrics |
| |
|
| | ### Core Metrics |
| |
|
| | - **Exact Match (EM)**: Percentage of queries where the predicted photo set exactly matches the ground truth |
| | - **F1 Score**: Harmonic mean of precision and recall at the set level |
| |
|
| | ### Query Types |
| |
|
| | 1. **Intra-Event**: Search within a single event or time period (e.g., "sunset photos from our beach vacation") |
| | 2. **Inter-Event**: Search across multiple events or time periods (e.g., "all birthday party photos from last year") |
| |
|
| | ### Tracks |
| |
|
| | - **Standard Track**: Uses predefined constraints and standard model configurations |
| | - **Open Track**: Allows custom models, additional training data, and external resources |
| |
|
| | All metrics are reported as: |
| | - Overall (all queries) |
| | - Intra-event only |
| | - Inter-event only |
| |
|
| | ## 🚀 How to Submit |
| |
|
| | ### Step-by-Step Guide |
| |
|
| | 1. **Prepare Your Results** |
| | - Run your method on the DISBench test set |
| | - Format predictions according to the submission schema (see below) |
| |
|
| | 2. **Submit via Web Interface** |
| | - Navigate to the **Submit** tab on this Space |
| | - Upload your JSON file containing metadata and predictions |
| | - Click "Submit" |
| |
|
| | 3. **Automated Processing** |
| | - The system validates your submission format |
| | - A Pull Request is automatically created |
| | - Maintainers review the submission |
| |
|
| | 4. **Leaderboard Update** |
| | - Once approved and merged, the Space automatically rebuilds |
| | - Your results are evaluated against ground truth |
| | - The leaderboard updates with your scores |
| |
|
| | ### Submission Format |
| |
|
| | ```json |
| | { |
| | "meta": { |
| | "method_name": "Your Method Name", |
| | "organization": "Your Organization", |
| | "track": "Standard", |
| | "agent_framework": "Your Agent Framework (if applicable)", |
| | "backbone_model": "Your Backbone Model", |
| | "retriever_model": "Your Retriever Model (if applicable)", |
| | "project_url": "https://github.com/your-repo" |
| | }, |
| | "predictions": { |
| | "1": ["photo_id_1", "photo_id_2", "photo_id_3"], |
| | "2": ["photo_id_4"], |
| | "3": ["photo_id_5", "photo_id_6"], |
| | ... |
| | } |
| | } |
| | ``` |
| |
|
| | ### Field Descriptions |
| |
|
| | **Meta Fields:** |
| | - `method_name` (required): Name of your method/system |
| | - `organization` (optional): Your institution or organization |
| | - `track` (required): Either "Standard" or "Open" |
| | - `agent_framework` (optional): Agent framework used (e.g., "ReAct", "AutoGPT") |
| | - `backbone_model` (required): Core model used (e.g., "GPT-4", "Claude-3") |
| | - `retriever_model` (optional): Retrieval model used (e.g., "CLIP-ViT-L/14", "BM25") |
| | - `project_url` (optional): Link to your project/paper |
| |
|
| | **Predictions:** |
| | - Keys are query IDs (as strings) |
| | - Values are arrays of photo IDs (as strings) |
| | - Photo IDs should match those in the ground truth dataset |
| |
|
| | ## 📋 Leaderboard Rules |
| |
|
| | ### Uniqueness & Deduplication |
| |
|
| | Each entry is uniquely identified by the combination of: |
| | - Method name |
| | - Agent framework |
| | - Backbone model |
| | - Retriever model |
| | - Track |
| |
|
| | If you submit multiple times with the same configuration, only the **latest submission** will appear on the leaderboard. |
| |
|
| | ### Ranking |
| |
|
| | Entries are ranked by **Overall EM Score** in descending order. The leaderboard displays: |
| | - Overall EM & F1 |
| | - Intra-event EM & F1 |
| | - Inter-event EM & F1 |
| |
|
| | ### Separate Tracks |
| |
|
| | Standard and Open track submissions are ranked separately to ensure fair comparison. |
| |
|
| | ## 📄 Citation |
| |
|
| | If you use DISBench in your research, please cite: |
| |
|
| | ```bibtex |
| | @misc{deng2026deepimagesearchbenchmarkingmultimodalagents, |
| | title={DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories}, |
| | author={Chenlong Deng and Mengjie Deng and Junjie Wu and Dun Zeng and Teng Wang and Qingsong Xie and Jiadeng Huang and Shengjie Ma and Changwang Zhang and Zhaoxiang Wang and Jun Wang and Yutao Zhu and Zhicheng Dou}, |
| | year={2026}, |
| | eprint={2602.10809}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CV}, |
| | url={https://arxiv.org/abs/2602.10809} |
| | } |
| | ``` |
| |
|
