README.md

---
license: apache-2.0
language:
  - en
pipeline_tag: image-text-to-text
tags:
  - agriculture
  - multimodal
  - vision-language
  - visual-grounding
  - image-segmentation
  - grounded-caption-generation
  - referring-expression-segmentation
  - sam2
  - bioclip
  - dinov3
inference: false
---

# AgriScope

<p align="center">
  <strong>Pixel-Grounded Multimodal Understanding for Agriculture Images</strong>
</p>

<p align="center">
  <strong>Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain, Sajid Javed</strong><br>
  Khalifa University of Science and Technology, Abu Dhabi, UAE
</p>

<p align="center">
  <a href="https://github.com/boudiafA/AgriScope"><img src="https://img.shields.io/badge/GitHub-Repository-181717" alt="GitHub repository"></a>
  <img src="https://img.shields.io/badge/Paper-Coming%20soon-lightgrey" alt="Paper coming soon">
  <img src="https://img.shields.io/badge/Weights-Coming%20soon-lightgrey" alt="Weights coming soon">
  <img src="https://img.shields.io/badge/Dataset-Coming%20soon-lightgrey" alt="Dataset coming soon">
</p>

> **Release status:** This repository currently provides the AgriScope model card, project figures, task definitions, and reported manuscript results. Model weights, configuration files, processors, training code, and AgriGround annotations are coming soon.

## Model Description

<p align="center">
  <img src="images/overview.png" alt="AgriScope and AgriGround overview" width="100%">
</p>

**AgriScope** is a pixel-grounded multimodal model for agricultural image understanding. It supports image-level, region-level, and pixel-level interaction within one framework, connecting generated agricultural concepts to segmentation masks and spatial annotations.

The model is designed to identify, describe, localize, and segment agricultural entities including plant diseases, lesions, pests, crops, weeds, botanical species, plant organs, and structural features. It supports both text-only responses and responses grounded in masks or normalized bounding boxes.

AgriScope is trained with **AgriGround**, a large-scale pixel-grounded agricultural instruction-tuning dataset containing **503,919 images** and **11,421,148 samples** across **14 tasks**.

### Model Details

| Property | Description |
|---|---|
| Model type | Pixel-grounded multimodal language model |
| Domain | Agriculture and plant imagery |
| Languages | English |
| Parameter count | 2B total parameters, as reported in the manuscript |
| Visual representation | Biological semantic and dense spatial encoders |
| Grounding | Language-conditioned `[SEG]` states with a SAM2-driven decoder |
| Adaptation | Projection alignment followed by LoRA instruction tuning |
| License | Apache-2.0 for repository materials and future code release |
| Model weights | **Coming soon** |

## Architecture

<p align="center">
  <img src="images/architecture.png" alt="AgriScope architecture" width="100%">
</p>

AgriScope combines complementary semantic and spatial pathways:

1. A biological contextual encoder extracts agricultural and biological semantics.
2. A global contextual encoder preserves texture, morphology, boundaries, and dense spatial information.
3. Projection layers map visual representations into the multimodal language space.
4. The language model generates text and special `[SEG]` tokens for grounded concepts.
5. Each `[SEG]` hidden state conditions the SAM2-driven mask decoder to produce a corresponding pixel-level mask.

The manuscript implementation initializes its visual and grounding components from BioCLIP, DINOv3, and SAM2. The large pretrained encoders remain frozen while alignment modules and LoRA parameters are optimized.

### Inputs

- An agricultural RGB image
- A natural-language instruction or question
- An optional region for region-conditioned tasks
- Optional conversation history for multi-turn interaction

### Outputs

- Natural-language captions and answers
- Interleaved grounded captions with phrase-mask correspondence
- Referring-expression, semantic, and part segmentation masks
- Region-conditioned descriptions and conversations
- Counts and normalized bounding boxes
- Multi-turn grounded responses

## Intended Uses

### Primary Research Uses

- Agricultural image captioning and visual question answering
- Plant disease, pest, weed, crop, and species understanding
- Referring expression and semantic segmentation
- Grounded caption generation
- Region-level and multi-turn grounded interaction
- Agricultural object counting, detection, and localization
- Research on interpretable and evidence-grounded agricultural AI

### Out-of-Scope Uses

- Autonomous pesticide, treatment, or crop-management decisions without expert review
- Safety-critical agricultural robotics without independent perception safeguards
- Regulatory, insurance, or legal determinations
- Identification of entities outside the supported visual and agricultural domains

## AgriGround Training Data

<p align="center">
  <img src="images/annotation_pipeline.png" alt="AgriGround annotation pipeline" width="100%">
</p>

AgriGround is produced using a four-stage annotation and task-generation pipeline:

1. Generate detailed image captions, class descriptions, counts, and bounding-box metadata.
2. Correct captions, identify grounded object phrases, and prepare segmentation prompts.
3. Generate and align segmentation masks with grounded phrases.
4. Synthesize instruction-following records for the 14 supported tasks.

### Dataset Statistics

| Split | Images | Samples | Average samples/image |
|---|---:|---:|---:|
| Train | 401,234 | 9,095,320 | 22.66 |
| Test | 102,685 | 2,325,828 | 22.66 |
| **Total** | **503,919** | **11,421,148** | **22.66** |

| Source group | Images | Share |
|---|---:|---:|
| Classification datasets | 232,923 | 46.22% |
| Detection datasets | 27,938 | 5.54% |
| iNatAg subset | 169,324 | 33.60% |
| Insects (IP102) | 73,734 | 14.63% |

### Supported Tasks

| Family | Tasks |
|---|---|
| Captioning | Image-level captioning, region-level captioning, grounded caption generation |
| Segmentation | Referring expression segmentation, semantic segmentation, part segmentation |
| Detection and localization | Phrase grounding, grounded counting, grounded detection, reasoning detection |
| Conversation and QA | Region-level conversation, multi-turn grounded conversation, classification QA, negative absence QA |

<p align="center">
  <img src="images/task_examples_v3.png" alt="Task-specific AgriGround examples" width="62%">
</p>

See [docs/TASKS.md](docs/TASKS.md) for definitions and per-task sample counts.

The source agricultural images are not distributed in this repository. Their original licenses and terms remain applicable. Annotation download instructions and dataset-specific licensing details will accompany the public dataset release.

## Training Procedure

Training follows two stages described in the manuscript:

1. **Visual-language and grounding alignment:** optimize projection layers, grounding modules, and the segmentation decoder while keeping the pretrained backbones frozen.
2. **Instruction tuning:** apply parameter-efficient LoRA adaptation using the 14 AgriGround tasks while retaining frozen visual encoders.

The joint objective combines autoregressive language modeling with binary cross-entropy and Dice losses for segmentation supervision. Full hyperparameters, preprocessing, and reproducibility scripts will be released with the code.

## Evaluation

The following results are reported in the current manuscript draft.

| Task | Metrics | AgriScope |
|---|---|---:|
| Image-level captioning | CIDEr / ASF | **146.4 / 86.7** |
| Region-level captioning | CIDEr / ASF | **132.5 / 84.8** |
| Classification QA | Accuracy / F1 | **82.4 / 80.7** |
| Grounded counting | Accuracy | **74.8** |
| Semantic segmentation | mIoU / Dice | **66.1 / 78.4** |
| Referring expression segmentation | J&F / cIoU | **67.30 / 72.65** |
| Grounded caption generation | METEOR / CIDEr | **27.9 / 118.6** |
| Grounded caption generation | AP50 / mIoU / Recall | **63.9 / 59.4 / 74.2** |

### Efficiency

| Parameters | GFLOPs | GPU memory | Inference time |
|---:|---:|---:|---:|
| **2B** | **177** | **6 GB** | **480 ms/image** |

Complete baseline comparisons, cross-dataset evaluation, ablations, and experimental settings will accompany the paper release.

## Qualitative Results

### Grounded Caption Generation

<p align="center">
  <img src="images/qualitative_gcg.png" alt="AgriScope grounded caption generation examples" width="100%">
</p>

### Representative Tasks

<p align="center">
  <img src="images/qualitative_tasks.png" alt="AgriScope representative task results" width="100%">
</p>

### Referring Expression Segmentation

<p align="center">
  <img src="images/referring_segmentation_comparison.png" alt="Referring expression segmentation comparison" width="100%">
</p>

## Limitations and Risks

- AgriScope can produce plausible but incorrect descriptions, classifications, counts, or masks.
- Performance may degrade under poor illumination, blur, occlusion, unusual viewpoints, severe domain shift, or very small targets.
- Fine-grained diseases, species, and pests with similar visual characteristics may be confused.
- Segmentation quality depends on the visual coverage and annotation quality of the training data.
- Dataset composition may encode geographic, crop, acquisition, and class-frequency biases from its source datasets.
- Outputs require review by qualified agricultural experts before being used for diagnosis or management decisions.

## Repository Contents

```text
.
|-- README.md
|-- CITATION.cff
|-- LICENSE
|-- docs/
|   `-- TASKS.md
`-- images/
    |-- overview.png
    |-- architecture.png
    |-- annotation_pipeline.png
    |-- task_examples_v3.png
    |-- qualitative_gcg.png
    |-- qualitative_tasks.png
    `-- referring_segmentation_comparison.png
```

## Release Roadmap

| Resource | Status |
|---|---|
| Model weights and configuration | **Coming soon** |
| Processor and inference example | **Coming soon** |
| Training and evaluation code | **Coming soon** |
| AgriGround train/test annotations | **Coming soon** |
| Paper and final citation | **Coming soon** |

Development updates and future code releases are tracked in the [AgriScope GitHub repository](https://github.com/boudiafA/AgriScope).

## Citation

The final paper link and citation will be added upon release. Until then, please use:

```bibtex
@misc{boudiaf2026agriscope,
  title  = {AgriScope: Pixel-Grounded Multimodal Understanding for Agriculture Images},
  author = {Boudiaf, Abderrahmene and Alanssari, Mohamad and Hussain, Irfan and Javed, Sajid},
  year   = {2026},
  note   = {Manuscript under review}
}
```

## License

Repository documentation and the future code release are provided under the [Apache License 2.0](LICENSE). AgriGround annotations and source images may be subject to separate terms, which will be documented with the dataset release.

## Acknowledgments

This work was conducted at Khalifa University of Science and Technology, Abu Dhabi, UAE. We acknowledge the creators and maintainers of the agricultural datasets and open-source foundation models that support this research.

## Contact

For questions and collaborations, use the [AgriScope GitHub issue tracker](https://github.com/boudiafA/AgriScope/issues) or the Hugging Face Community tab.