README.md

metadata

title: PhotoBench-Protected
emoji: 🛡️
colorFrom: green
colorTo: green
sdk: gradio
app_file: app.py
pinned: true
license: apache-2.0
short_description: PhotoBench-Protected agent-only leaderboard
tags:
  - leaderboard
  - image-retrieval
  - benchmark
  - agent

PhotoBench-Protected Leaderboard

Agent-only leaderboard for PhotoBench-Protected, where only partial captions, embeddings, and metadata are provided.

Quick Start

1. Download the protected dataset from PhotoBench-Protected Dataset
2. Build your agent-based retrieval system using the provided features
3. Submit predictions in JSON format

Important Notice

PhotoBench-Protected is our initial open-source release with limited information sources. This benchmark focuses exclusively on agent planning ability.

• For unrestricted retrieval with raw images, please use the full PhotoBench Leaderboard.
• The test sets for PhotoBench-Protected and PhotoBench (full) are different.
• Please confirm you are submitting to the correct leaderboard before uploading.

Evaluation Metrics

• Recall@k for k ∈ {1, 5, 10, 20, 50, 100}
• NDCG@k for k ∈ {1, 5, 10, 20, 50, 100}