README.md

---
license: mit
dataset_info:
  features:
  - name: image
    dtype: image
  - name: question_id
    dtype: string
  - name: question_type
    dtype: string
  - name: question
    dtype: string
  - name: A
    dtype: string
  - name: B
    dtype: string
  - name: C
    dtype: string
  - name: D
    dtype: string
  - name: answer
    dtype: string
  splits:
  - name: test
    num_bytes: 2559246769
    num_examples: 13950
  download_size: 1534670685
  dataset_size: 2559246769
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
extra_gated_fields:
  Full Name: text
  Email Address: text
  Institution or Company: text
  Country: country
  Intended Use of the Dataset:
    type: select
    options:
    - Academic Research
    - Educational Purpose
  I agree to use this dataset for non-commercial purposes only: checkbox
task_categories:
- visual-question-answering
language:
- en
tags:
- plant
- vqa
pretty_name: LeafBench
size_categories:
- 10K<n<100K
---

# LeafBench 🌿

**LeafBench** is a comprehensive visual question answering (VQA) benchmark derived from the [LeafNet](https://huggingface.co/collections/enalis/leafsight) dataset, designed to systematically evaluate the capabilities of Vision-Language Models (VLMs) in understanding plant diseases. It supports six hierarchical diagnostic tasks — from binary health screening to expert-level taxonomic reasoning.

> 📄 Paper: [LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases](https://arxiv.org/abs/2602.13662)  
> 🔗 Collection: [LeafSight on Hugging Face](https://huggingface.co/collections/enalis/leafsight)  
> 💻 Code: [github.com/EnalisUs/LeafBench](https://github.com/EnalisUs/LeafBench)

---

## Dataset Summary

LeafBench is a curated subset of LeafNet built specifically for evaluating large vision-language models on plant pathology tasks. Rather than relying on open-ended generation (which is prone to hallucination), LeafBench employs a **label-constrained prompting strategy** where models must select from a closed-world candidate set. This ensures reliable output parsing and consistent, context-bounded inference.

The benchmark includes:
- **Full (All)**: 2,910 images and **13,950 question-answer pairs**
- **Tiny subset**: 890 samples — a cost-effective proxy for evaluating commercial APIs or establishing human expert baselines

---

## Supported Tasks

LeafBench evaluates models across **six hierarchical diagnostic tasks**:

| Task | Abbreviation | Description |
|---|---|---|
| Healthy-Diseased Classification | HDC | Binary assessment of pathology presence |
| Disease Identification | DI | Identification of specific disease conditions |
| Pathogen Classification | PC | High-level causal agent (Fungal, Bacterial, Viral, Mite) |
| Crop Species Identification | CSI | Host plant species recognition across 22 species |
| Scientific Name Classification | SNC | Binomial-level scientific pathogen naming |
| Symptom Identification | SI | Fine-grained visual symptom recognition (lesion morphology, chlorosis, etc.) |

All questions are **visually dependent** — correct answers cannot be derived from text alone and require analysis of the accompanying leaf image.

---

## Dataset Structure

Each sample in LeafBench contains:

```python
{
  "image": "<PIL.Image>",          # RGB leaf image (JPG)
  "question": "<str>",             # Multiple-choice diagnostic question
  "choices": ["A. ...", "B. ...", "C. ...", "D. ..."],  # Candidate answer options
  "answer": "<str>",               # Ground-truth answer label (e.g., "A")
  "task": "<str>",                 # Task type: HDC | DI | PC | CSI | SNC | SI
  "species": "<str>",              # Crop species (e.g., "Apple", "Tomato")
  "disease": "<str>",              # Disease name (e.g., "Black Rot")
  "pathogenic_agent": "<str>",     # Pathogen category (e.g., "Fungal")
  "taxonomic_name": "<str>",       # Scientific name (e.g., "Botryosphaeria spp.")
  "symptom": "<str>",              # Symptom description (e.g., "Dark brown spots")
  "acquisition": "<str>"           # Image source: "Farm" or "Lab"
}
```

### Data Splits

| Split | Images | QA Pairs |
|---|---|---|
| All (Full Benchmark) | 2,910 | 13,950 |
| Tiny (Subset) | ~190 | 890 |

### QA Distribution by Task

| Task | Sample Count |
|---|---|
| Healthy-Diseased Classification (HDC) | ~2,325 |
| Disease Identification (DI) | ~2,325 |
| Crop Species Identification (CSI) | ~2,325 |
| Scientific Name Classification (SNC) | ~2,325 |
| Pathogen Classification (PC) | ~2,325 |
| Symptom Identification (SI) | ~2,325 |

---

## Dataset Creation

### Source Data

LeafBench is derived from **LeafNet**, a large-scale dataset comprising:
- **186,000** expert-annotated leaf images
- **22** common crop species
- **62** disease categories (97 total fine-grained classes including healthy controls)
- Images collected from **7 countries** across 3 continents (USA, India, Bangladesh, Kenya, Ghana, Tanzania, Vietnam)

### Curation Process

1. **Metadata synthesis**: Biological taxonomies (species, disease, pathogenic agent, symptom descriptions) were sourced from authoritative repositories (NIH, USDA NIFA).
2. **Expert verification**: All image-metadata pairs underwent review by agricultural domain experts to filter mislabeled or noisy samples.
3. **LeafBench extraction**: A targeted subset was selected from LeafNet to represent the full range of diagnostic task difficulty.
4. **Label-constrained QA construction**: Questions were formulated as closed-form multiple-choice items requiring visual-dependent reasoning.

### Annotation

- All annotations are **expert-curated**, not synthetically generated.
- Metadata fields include: species, disease name, pathogenic agent, taxonomic nomenclature, symptom description, image acquisition environment (Farm/Lab), and resolution.

---

## Evaluation

### Metric

**Accuracy (Acc)** is the primary evaluation metric — the percentage of questions correctly answered by comparing model predictions to ground-truth labels.

### Benchmark Results (Zero-Shot VQA)

| Model | HDC | DI | CSI | SNC | PC | SI | Avg. |
|---|---|---|---|---|---|---|---|
| GPT-4o | 92.48 | 85.27 | 85.58 | 65.27 | 56.47 | 51.64 | 72.78 |
| Gemini 2.5 Pro | 88.25 | 78.54 | 83.21 | 64.89 | 51.23 | 48.99 | 69.18 |
| SCOLD (domain-specific) | **96.28** | **95.85** | 84.73 | 41.64 | 37.83 | **77.92** | **72.38** |
| Qwen 2.5 VL | 81.05 | 52.60 | 63.06 | 41.50 | 60.92 | 43.14 | 57.04 |
| LLaVA-NeXT | 88.33 | 33.64 | 48.82 | 27.10 | 70.82 | 32.09 | 50.13 |
| BLIP-2 | 62.36 | 48.49 | 64.15 | 28.02 | 54.88 | 31.59 | 48.25 |
| CLIP | 21.20 | 46.51 | 48.99 | 32.56 | 20.43 | 32.32 | 33.67 |
| Random Baseline | 50.47 | 24.07 | 26.20 | 25.51 | 26.14 | 26.18 | ~29.8 |

> Results are reported on the full **All** benchmark under zero-shot evaluation.

### Key Findings

- **Binary HDC** tasks are easiest, with top models exceeding **90% accuracy**.
- **Fine-grained tasks** (SNC, SI, PC) are significantly harder — even frontier models struggle below **65%** on SNC.
- **Domain-specific models** (SCOLD) outperform general VLMs on disease and symptom tasks but struggle with taxonomic reasoning.
- Generic open-source VLMs (CLIP, SigLIP2) frequently perform near **random chance** on fine-grained tasks.

---

## Limitations

- Geographic coverage, while spanning 7 countries, could be broader to ensure global transferability.
- Images are static RGB captures and do not capture **temporal dynamics** (disease progression over time).
- Text annotations are limited to disease names, pathogen taxonomy, and brief symptom descriptions — **severity scales and growth stage metadata** are not included.
- Models trained or evaluated on LeafBench may not generalize to novel field settings with phenotypic expressions outside the visible spectrum.

---

## Citation

If you use LeafBench in your research, please cite:

```bibtex
@article{leafnet2026,
  title     = {LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for 
               Foundational Vision-Language Understanding of Plant Diseases},
  author    = {Khang Nguyen Quoc and Phuong D. Dao and Luyl-Da Quach},
  journal   = {arXiv preprint arXiv:2602.13662},
  year      = {2026}
}
```

---

## License

This dataset is released under the **Creative Commons Attribution 4.0 International (CC BY 4.0)** license.

---

## Contact

For questions or contributions, please open an issue on [GitHub](https://github.com/EnalisUs/LeafBench) or contact the authors via the paper's corresponding email.