AgriScope

Pixel-Grounded Multimodal Understanding for Agriculture Images

Abderrahmene Boudiaf, Mohamad Alanssari, Irfan Hussain, Sajid Javed
Khalifa University of Science and Technology, Abu Dhabi, UAE

Release status: This repository currently provides the AgriScope model card, project figures, task definitions, and reported manuscript results. Model weights, configuration files, processors, training code, and AgriGround annotations are coming soon.

Model Description

AgriScope is a pixel-grounded multimodal model for agricultural image understanding. It supports image-level, region-level, and pixel-level interaction within one framework, connecting generated agricultural concepts to segmentation masks and spatial annotations.

The model is designed to identify, describe, localize, and segment agricultural entities including plant diseases, lesions, pests, crops, weeds, botanical species, plant organs, and structural features. It supports both text-only responses and responses grounded in masks or normalized bounding boxes.

AgriScope is trained with AgriGround, a large-scale pixel-grounded agricultural instruction-tuning dataset containing 503,919 images and 11,421,148 samples across 14 tasks.

Model Details

Property	Description
Model type	Pixel-grounded multimodal language model
Domain	Agriculture and plant imagery
Languages	English
Parameter count	2B total parameters, as reported in the manuscript
Visual representation	Biological semantic and dense spatial encoders
Grounding	Language-conditioned [SEG] states with a SAM2-driven decoder
Adaptation	Projection alignment followed by LoRA instruction tuning
License	Apache-2.0 for repository materials and future code release
Model weights	Coming soon

Architecture

AgriScope combines complementary semantic and spatial pathways:

1. A biological contextual encoder extracts agricultural and biological semantics.
2. A global contextual encoder preserves texture, morphology, boundaries, and dense spatial information.
3. Projection layers map visual representations into the multimodal language space.
4. The language model generates text and special [SEG] tokens for grounded concepts.
5. Each [SEG] hidden state conditions the SAM2-driven mask decoder to produce a corresponding pixel-level mask.

The manuscript implementation initializes its visual and grounding components from BioCLIP, DINOv3, and SAM2. The large pretrained encoders remain frozen while alignment modules and LoRA parameters are optimized.

Inputs

• An agricultural RGB image
• A natural-language instruction or question
• An optional region for region-conditioned tasks
• Optional conversation history for multi-turn interaction

Outputs

• Natural-language captions and answers
• Interleaved grounded captions with phrase-mask correspondence
• Referring-expression, semantic, and part segmentation masks
• Region-conditioned descriptions and conversations
• Counts and normalized bounding boxes
• Multi-turn grounded responses

Intended Uses

Primary Research Uses

• Agricultural image captioning and visual question answering
• Plant disease, pest, weed, crop, and species understanding
• Referring expression and semantic segmentation
• Grounded caption generation
• Region-level and multi-turn grounded interaction
• Agricultural object counting, detection, and localization
• Research on interpretable and evidence-grounded agricultural AI

Out-of-Scope Uses

• Autonomous pesticide, treatment, or crop-management decisions without expert review
• Safety-critical agricultural robotics without independent perception safeguards
• Regulatory, insurance, or legal determinations
• Identification of entities outside the supported visual and agricultural domains

AgriGround Training Data

AgriGround is produced using a four-stage annotation and task-generation pipeline:

1. Generate detailed image captions, class descriptions, counts, and bounding-box metadata.
2. Correct captions, identify grounded object phrases, and prepare segmentation prompts.
3. Generate and align segmentation masks with grounded phrases.
4. Synthesize instruction-following records for the 14 supported tasks.

Dataset Statistics

Split	Images	Samples	Average samples/image
Train	401,234	9,095,320	22.66
Test	102,685	2,325,828	22.66
Total	503,919	11,421,148	22.66

Source group	Images	Share
Classification datasets	232,923	46.22%
Detection datasets	27,938	5.54%
iNatAg subset	169,324	33.60%
Insects (IP102)	73,734	14.63%

Supported Tasks

Family	Tasks
Captioning	Image-level captioning, region-level captioning, grounded caption generation
Segmentation	Referring expression segmentation, semantic segmentation, part segmentation
Detection and localization	Phrase grounding, grounded counting, grounded detection, reasoning detection
Conversation and QA	Region-level conversation, multi-turn grounded conversation, classification QA, negative absence QA

See docs/TASKS.md for definitions and per-task sample counts.

The source agricultural images are not distributed in this repository. Their original licenses and terms remain applicable. Annotation download instructions and dataset-specific licensing details will accompany the public dataset release.

Training Procedure

Training follows two stages described in the manuscript:

1. Visual-language and grounding alignment: optimize projection layers, grounding modules, and the segmentation decoder while keeping the pretrained backbones frozen.
2. Instruction tuning: apply parameter-efficient LoRA adaptation using the 14 AgriGround tasks while retaining frozen visual encoders.

The joint objective combines autoregressive language modeling with binary cross-entropy and Dice losses for segmentation supervision. Full hyperparameters, preprocessing, and reproducibility scripts will be released with the code.

Evaluation

The following results are reported in the current manuscript draft.

Task	Metrics	AgriScope
Image-level captioning	CIDEr / ASF	146.4 / 86.7
Region-level captioning	CIDEr / ASF	132.5 / 84.8
Classification QA	Accuracy / F1	82.4 / 80.7
Grounded counting	Accuracy	74.8
Semantic segmentation	mIoU / Dice	66.1 / 78.4
Referring expression segmentation	J&F / cIoU	67.30 / 72.65
Grounded caption generation	METEOR / CIDEr	27.9 / 118.6
Grounded caption generation	AP50 / mIoU / Recall	63.9 / 59.4 / 74.2

Efficiency

Parameters	GFLOPs	GPU memory	Inference time
2B	177	6 GB	480 ms/image

Complete baseline comparisons, cross-dataset evaluation, ablations, and experimental settings will accompany the paper release.

Qualitative Results

Grounded Caption Generation

Representative Tasks

Referring Expression Segmentation

Limitations and Risks

• AgriScope can produce plausible but incorrect descriptions, classifications, counts, or masks.
• Performance may degrade under poor illumination, blur, occlusion, unusual viewpoints, severe domain shift, or very small targets.
• Fine-grained diseases, species, and pests with similar visual characteristics may be confused.
• Segmentation quality depends on the visual coverage and annotation quality of the training data.
• Dataset composition may encode geographic, crop, acquisition, and class-frequency biases from its source datasets.
• Outputs require review by qualified agricultural experts before being used for diagnosis or management decisions.

Repository Contents

.
|-- README.md
|-- CITATION.cff
|-- LICENSE
|-- docs/
|   `-- TASKS.md
`-- images/
    |-- overview.png
    |-- architecture.png
    |-- annotation_pipeline.png
    |-- task_examples_v3.png
    |-- qualitative_gcg.png
    |-- qualitative_tasks.png
    `-- referring_segmentation_comparison.png

Release Roadmap

Resource	Status
Model weights and configuration	Coming soon
Processor and inference example	Coming soon
Training and evaluation code	Coming soon
AgriGround train/test annotations	Coming soon
Paper and final citation	Coming soon

Development updates and future code releases are tracked in the AgriScope GitHub repository.

Citation

The final paper link and citation will be added upon release. Until then, please use:

@misc{boudiaf2026agriscope,
  title  = {AgriScope: Pixel-Grounded Multimodal Understanding for Agriculture Images},
  author = {Boudiaf, Abderrahmene and Alanssari, Mohamad and Hussain, Irfan and Javed, Sajid},
  year   = {2026},
  note   = {Manuscript under review}
}

License

Repository documentation and the future code release are provided under the Apache License 2.0. AgriGround annotations and source images may be subject to separate terms, which will be documented with the dataset release.

Acknowledgments

This work was conducted at Khalifa University of Science and Technology, Abu Dhabi, UAE. We acknowledge the creators and maintainers of the agricultural datasets and open-source foundation models that support this research.

Contact

For questions and collaborations, use the AgriScope GitHub issue tracker or the Hugging Face Community tab.