Blame for README.md

---
license: apache-2.0
base_model: vllm-sr/d3-flash
base_model_relation: finetune
library_name: transformers
pipeline_tag: zero-shot-classification
language:
- en
tags:
- earth-observation
- remote-sensing
- satellite-imagery
- decision-model
- multimodal
- vision
- calibration
- lora
---

# EO-Jev: Decision Model for Earth Observation

[**Paper (arXiv:2611.03371)**](https://arxiv.org/abs/2611.03371) ·
[**Project page**](https://mbzuai-oryx.github.io/EO-Jev/) ·
[**Code**](https://github.com/mbzuai-oryx/EO-Jev)

EO-Jev answers questions about satellite and aerial images with **a probability for every allowed answer, in a single
forward pass**. It writes no text, so answers come fast and with a confidence you can threshold: accept confident
answers automatically and send uncertain ones to an analyst.

EO-Jev is [d3-flash](https://huggingface.co/vllm-sr/d3-flash) (8.4B parameters, built on Qwen3.5-9B) adapted to Earth
observation. We trained LoRA adapters on public Earth-observation training splits: rank 32 on the text tower, 64 on the
vision merger and 16 on the vision blocks. The adapters are merged into the weights, so EO-Jev has the same
architecture, interface and inference cost as d3-flash.

## Results

On 30 evaluation settings from 23 public Earth-observation benchmarks (details in the paper):

| | |
|---|---|
| **26 / 30** | settings improved significantly over d3-flash: +23.8 points on average where the dataset's training split was used, **+8.0** on 12 settings held out from adaptation |
| **20 / 28** | settings at or above the best open Earth-observation specialist model |
| **21 / 28** | settings with lower expected calibration error than the specialist |
| **6.4×** | median throughput of the best specialist's, each at its best batch size (3.2× with one question per call) |
| **−1.8** | points on general (non-EO) decisions, the cost of adaptation |

## Quickstart

```bash
pip install "transformers==5.17.0" torch torchvision pillow safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers
pip install causal-conv1d           # optional: fast kernel for the short convolutions in those layers
```

```python
import json

from huggingface_hub import hf_hub_download
from transformers import AutoModel

model = AutoModel.from_pretrained("mx297/EO-Jev", trust_remote_code=True)  # on cuda:0 when a GPU is visible

image = hf_hub_download("mx297/EO-Jev", "assets/example_airport.png")  # or a local path, URL or PIL image
result = model.system_one(
    state="Aerial image, about 0.3 m per pixel.",
    images=[image],
    questions={
        "scene": {
            "type": "choice",
            "instructions": "What is the main land use in this image?",
            "criteria": {"airport": None, "harbor": None, "parking lot": None, "residential area": None, "farmland": None},
        },
        "airplanes": {
            "type": "choice",
            "instructions": "How many airplanes are in this image?",
            "criteria": {"0": None, "1": None, "2": None, "3": None, "4": None, "5": None, "more than 5": None},
        },
        "car": {
            "type": "noul",
            "instructions": "Is there a car in this image?",
        },
    },
)
print(json.dumps(result["answers"], indent=1))
```

Output (probabilities rounded to three decimals):

```json
{
 "scene": {"type": "choice", "choice": "airport",
           "probabilities": {"airport": 0.966, "harbor": 0.000, "parking lot": 0.033, "residential area": 0.000, "farmland": 0.000},
           "confidence": 0.905},
 "airplanes": {"type": "choice", "choice": "5",
               "probabilities": {"0": 0.000, "1": 0.001, "2": 0.002, "3": 0.046, "4": 0.440, "5": 0.454, "more than 5": 0.057},
               "confidence": 0.463},
 "car": {"type": "noul", "noul": 0.978}
}
```

The scene and the car are answered right and with confidence. The image has four airplanes: EO-Jev's count is a near
tie between 4 (0.440) and 5 (0.454), and its low confidence (0.46) marks the answer for review instead of hiding the
mistake. Counting on unfamiliar imagery is one of the limitations below. For `noul` questions, the value is the
probability of "yes".

All questions about the same image or images are answered in one call.

## Asking questions

A request has a **state** (free text: what the image is, its sensor or resolution, any context), one or more
**images**, and a dictionary of **questions**. Each question has a `type`:

| Type | `criteria` | Answer |
|---|---|---|
| `choice` | a dict of options (up to 255); each value is an optional description (`None` is fine) | `choice` (the top option), `probabilities` per option, `confidence` |
| `noul` | none (a yes/no question) | `noul`: the probability of "yes" |
| `score` | an ordered list of 2–10 levels, e.g. `["no damage", "minor damage", "major damage", "destroyed"]` | `score` (the expected level index) and `probabilities` per level |

Several images go into one request, for example a pre- and post-event pair or a time series. Say in the state which is
which:

```python
result = model.system_one(
    state="Image 1 is the earlier (pre-disaster) image and Image 2 the later (post-disaster) image of the same area.",
    images=["pre.png", "post.png"],
    questions={
        "damage": {
            "type": "score",
            "instructions": "How damaged is the building in the centre?",
            "criteria": ["no damage", "minor damage", "major damage", "destroyed"],
        }
    },
)
```

The model also works as a `transformers` pipeline:
`pipeline("decision", model="mx297/EO-Jev", trust_remote_code=True)(state=..., questions=..., images=...)`.

**Tips**
- Keep each answer's wording short and distinct. The probabilities are over the options you give, so include an
  option such as "none of these" or "0" when it can be the answer.
- The probabilities are calibrated at the checkpoint's temperature. A per-question threshold on the top probability
  decides which answers to accept automatically.
- The code that runs the model (`d3_runtime.py` and the other `d3_*.py` files) is d3-flash's runtime, unchanged.

## Training

- **Base model:** d3-flash, with LoRA adapters on the text tower, the vision merger and the vision blocks (the vision
  blocks at 0.2× the learning rate), merged after training.
- **Data:** decision questions built from public Earth-observation training splits: scene and land-use labels, object
  presence and counting, visual question answering, multi-temporal change, building damage and disaster type,
  geo-localisation. About 20% of every batch replays general decision data with d3-flash's own probabilities as targets,
  to keep its general skills. Every training image that duplicates or crops an evaluation image was removed.
- **Loss:** KL divergence to the target distribution: one-hot, soft ordinal labels (SORD) for counts and damage
  levels, or d3-flash's own probabilities.
- **Compute:** 20,000 steps of 16 questions on 4 NVIDIA A100 GPUs.

Training and evaluation code: [github.com/mbzuai-oryx/EO-Jev](https://github.com/mbzuai-oryx/EO-Jev).

## Limitations

- Counting on unfamiliar imagery, robustness to the order of answer options, geometric and cross-view reasoning, and
  calibration far from the training data remain the hardest cases.
- EO-Jev was trained and evaluated on optical (RGB) imagery; SAR and multispectral inputs were not part of training.
- It answers only with the options you provide. It cannot say that none applies unless such an option is offered.
- Validate it on your own data before relying on it for decisions with consequences.

## License

Apache-2.0 ([LICENSE](LICENSE)), as d3-flash, which is built on
[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Apache-2.0); see [NOTICE](NOTICE). The datasets used for
training keep their own licences. The example image is from the UC Merced Land Use dataset (USGS National Map imagery,
public domain).

## Citation

```bibtex
@article{abouelhadid2026eojev,
  title   = {{EO-Jev}: Decision Model for {Earth} Observation},
  author  = {Abouelhadid, Mohamed Atef and Abuhani, Diaa Addeen and
             Munir, Muhammad Akhtar and Khan, Fahad Shahbaz and Khan, Salman},
  journal = {arXiv preprint arXiv:2611.03371},
  year    = {2026}
}
```

Please also cite d3-flash:

```bibtex
@misc{d3_flash_2026,
  title        = {{d3-flash}: A Multimodal Foundation Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-flash}}
}
```