EO-Jev / README.md

Model card: the quickstart's real output (run from the Hub on an A100)

75e5101 verified about 24 hours ago

8.6 kB

---
license: apache-2.0
base_model: vllm-sr/d3-flash
base_model_relation: finetune
library_name: transformers
pipeline_tag: zero-shot-classification
language:
- en
tags:
- earth-observation
- remote-sensing
- satellite-imagery
- decision-model
- multimodal
- vision
- calibration
- lora
---
# EO-Jev: Decision Model for Earth Observation

[**Paper (arXiv:2611.03371)**](https://arxiv.org/abs/2611.03371) ·
[**Project page**](https://mbzuai-oryx.github.io/EO-Jev/) ·
[**Code**](https://github.com/mbzuai-oryx/EO-Jev)

EO-Jev answers questions about satellite and aerial images with **a probability for every allowed answer, in a single
forward pass**. It writes no text, so answers come fast and with a confidence you can threshold: accept confident
answers automatically and send uncertain ones to an analyst.

EO-Jev is [d3-flash](https://huggingface.co/vllm-sr/d3-flash) (8.4B parameters, built on Qwen3.5-9B) adapted to Earth
observation. We trained LoRA adapters on public Earth-observation training splits: rank 32 on the text tower, 64 on the
vision merger and 16 on the vision blocks. The adapters are merged into the weights, so EO-Jev has the same
architecture, interface and inference cost as d3-flash.

## Results

On 30 evaluation settings from 23 public Earth-observation benchmarks (details in the paper):

| | |
|---|---|
| **26 / 30** | settings improved significantly over d3-flash: +23.8 points on average where the dataset's training split was used, **+8.0** on 12 settings held out from adaptation |
| **20 / 28** | settings at or above the best open Earth-observation specialist model |
| **21 / 28** | settings with lower expected calibration error than the specialist |
| **6.4×** | median throughput of the best specialist's, each at its best batch size (3.2× with one question per call) |
| **−1.8** | points on general (non-EO) decisions, the cost of adaptation |

## Quickstart

```bash
pip install "transformers==5.17.0" torch torchvision pillow safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers
pip install causal-conv1d           # optional: fast kernel for the short convolutions in those layers
```

```python
import json
from huggingface_hub import hf_hub_download
from transformers import AutoModel
model = AutoModel.from_pretrained("mx297/EO-Jev", trust_remote_code=True)  # on cuda:0 when a GPU is visible
image = hf_hub_download("mx297/EO-Jev", "assets/example_airport.png")  # or a local path, URL or PIL image
result = model.system_one(
    state="Aerial image, about 0.3 m per pixel.",
    images=[image],
    questions={
        "scene": {
            "type": "choice",
            "instructions": "What is the main land use in this image?",
            "criteria": {"airport": None, "harbor": None, "parking lot": None, "residential area": None, "farmland": None},
        },
        "airplanes": {
            "type": "choice",
            "instructions": "How many airplanes are in this image?",
            "criteria": {"0": None, "1": None, "2": None, "3": None, "4": None, "5": None, "more than 5": None},
        },
        "car": {
            "type": "noul",
            "instructions": "Is there a car in this image?",
        },
    },
)
print(json.dumps(result["answers"], indent=1))
```

Output (probabilities rounded to three decimals):

```json
{
 "scene": {"type": "choice", "choice": "airport",
           "probabilities": {"airport": 0.966, "harbor": 0.000, "parking lot": 0.033, "residential area": 0.000, "farmland": 0.000},
           "confidence": 0.905},
 "airplanes": {"type": "choice", "choice": "5",
               "probabilities": {"0": 0.000, "1": 0.001, "2": 0.002, "3": 0.046, "4": 0.440, "5": 0.454, "more than 5": 0.057},
               "confidence": 0.463},
 "car": {"type": "noul", "noul": 0.978}
}
```

The scene and the car are answered right and with confidence. The image has four airplanes: EO-Jev's count is a near
tie between 4 (0.440) and 5 (0.454), and its low confidence (0.46) marks the answer for review instead of hiding the
mistake. Counting on unfamiliar imagery is one of the limitations below. For `noul` questions, the value is the
probability of "yes".

All questions about the same image or images are answered in one call.

## Asking questions

A request has a **state** (free text: what the image is, its sensor or resolution, any context), one or more
**images**, and a dictionary of **questions**. Each question has a `type`:

| Type | `criteria` | Answer |
|---|---|---|
| `choice` | a dict of options (up to 255); each value is an optional description (`None` is fine) | `choice` (the top option), `probabilities` per option, `confidence` |
| `noul` | none (a yes/no question) | `noul`: the probability of "yes" |
| `score` | an ordered list of 2–10 levels, e.g. `["no damage", "minor damage", "major damage", "destroyed"]` | `score` (the expected level index) and `probabilities` per level |

Several images go into one request, for example a pre- and post-event pair or a time series. Say in the state which is
which:

```python
result = model.system_one(
    state="Image 1 is the earlier (pre-disaster) image and Image 2 the later (post-disaster) image of the same area.",
    images=["pre.png", "post.png"],
    questions={
        "damage": {
            "type": "score",
            "instructions": "How damaged is the building in the centre?",
            "criteria": ["no damage", "minor damage", "major damage", "destroyed"],
        }
    },
)
```

The model also works as a `transformers` pipeline:
`pipeline("decision", model="mx297/EO-Jev", trust_remote_code=True)(state=..., questions=..., images=...)`.

**Tips**
- Keep each answer's wording short and distinct. The probabilities are over the options you give, so include an
  option such as "none of these" or "0" when it can be the answer.
- The probabilities are calibrated at the checkpoint's temperature. A per-question threshold on the top probability
  decides which answers to accept automatically.
- The code that runs the model (`d3_runtime.py` and the other `d3_*.py` files) is d3-flash's runtime, unchanged.

## Training

- **Base model:** d3-flash, with LoRA adapters on the text tower, the vision merger and the vision blocks (the vision
  blocks at 0.2× the learning rate), merged after training.
- **Data:** decision questions built from public Earth-observation training splits: scene and land-use labels, object
  presence and counting, visual question answering, multi-temporal change, building damage and disaster type,
  geo-localisation. About 20% of every batch replays general decision data with d3-flash's own probabilities as targets,
  to keep its general skills. Every training image that duplicates or crops an evaluation image was removed.
- **Loss:** KL divergence to the target distribution: one-hot, soft ordinal labels (SORD) for counts and damage
  levels, or d3-flash's own probabilities.
- **Compute:** 20,000 steps of 16 questions on 4 NVIDIA A100 GPUs.

Training and evaluation code: [github.com/mbzuai-oryx/EO-Jev](https://github.com/mbzuai-oryx/EO-Jev).

## Limitations

- Counting on unfamiliar imagery, robustness to the order of answer options, geometric and cross-view reasoning, and
  calibration far from the training data remain the hardest cases.
- EO-Jev was trained and evaluated on optical (RGB) imagery; SAR and multispectral inputs were not part of training.
- It answers only with the options you provide. It cannot say that none applies unless such an option is offered.
- Validate it on your own data before relying on it for decisions with consequences.

## License

Apache-2.0 ([LICENSE](LICENSE)), as d3-flash, which is built on
[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Apache-2.0); see [NOTICE](NOTICE). The datasets used for
training keep their own licences. The example image is from the UC Merced Land Use dataset (USGS National Map imagery,
public domain).

## Citation

```bibtex
@article{abouelhadid2026eojev,
  title   = {{EO-Jev}: Decision Model for {Earth} Observation},
  author  = {Abouelhadid, Mohamed Atef and Abuhani, Diaa Addeen and
             Munir, Muhammad Akhtar and Khan, Fahad Shahbaz and Khan, Salman},
  journal = {arXiv preprint arXiv:2611.03371},
  year    = {2026}
}
```

Please also cite d3-flash:

```bibtex
@misc{d3_flash_2026,
  title        = {{d3-flash}: A Multimodal Foundation Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-flash}}
}
```

	---
	license: apache-2.0
	base_model: vllm-sr/d3-flash
	base_model_relation: finetune
	library_name: transformers
	pipeline_tag: zero-shot-classification
	language:
	- en
	tags:
	- earth-observation
	- remote-sensing
	- satellite-imagery
	- decision-model
	- multimodal
	- vision
	- calibration
	- lora
	---

	# EO-Jev: Decision Model for Earth Observation

	[Paper (arXiv:2611.03371)](https://arxiv.org/abs/2611.03371) ·
	[Project page](https://mbzuai-oryx.github.io/EO-Jev/) ·
	[Code](https://github.com/mbzuai-oryx/EO-Jev)

	EO-Jev answers questions about satellite and aerial images with **a probability for every allowed answer, in a single
	forward pass**. It writes no text, so answers come fast and with a confidence you can threshold: accept confident
	answers automatically and send uncertain ones to an analyst.

	EO-Jev is [d3-flash](https://huggingface.co/vllm-sr/d3-flash) (8.4B parameters, built on Qwen3.5-9B) adapted to Earth
	observation. We trained LoRA adapters on public Earth-observation training splits: rank 32 on the text tower, 64 on the
	vision merger and 16 on the vision blocks. The adapters are merged into the weights, so EO-Jev has the same
	architecture, interface and inference cost as d3-flash.

	## Results

	On 30 evaluation settings from 23 public Earth-observation benchmarks (details in the paper):

	\| \| \|
	\|---\|---\|
	\| 26 / 30 \| settings improved significantly over d3-flash: +23.8 points on average where the dataset's training split was used, +8.0 on 12 settings held out from adaptation \|
	\| 20 / 28 \| settings at or above the best open Earth-observation specialist model \|
	\| 21 / 28 \| settings with lower expected calibration error than the specialist \|
	\| 6.4× \| median throughput of the best specialist's, each at its best batch size (3.2× with one question per call) \|
	\| −1.8 \| points on general (non-EO) decisions, the cost of adaptation \|

	## Quickstart

	```bash
	pip install "transformers==5.17.0" torch torchvision pillow safetensors accelerate
	pip install flash-linear-attention # optional: fast GPU kernels for the linear-attention layers
	pip install causal-conv1d # optional: fast kernel for the short convolutions in those layers
	```

	```python
	import json

	from huggingface_hub import hf_hub_download
	from transformers import AutoModel

	model = AutoModel.from_pretrained("mx297/EO-Jev", trust_remote_code=True) # on cuda:0 when a GPU is visible

	image = hf_hub_download("mx297/EO-Jev", "assets/example_airport.png") # or a local path, URL or PIL image
	result = model.system_one(
	state="Aerial image, about 0.3 m per pixel.",
	images=[image],
	questions={
	"scene": {
	"type": "choice",
	"instructions": "What is the main land use in this image?",
	"criteria": {"airport": None, "harbor": None, "parking lot": None, "residential area": None, "farmland": None},
	},
	"airplanes": {
	"type": "choice",
	"instructions": "How many airplanes are in this image?",
	"criteria": {"0": None, "1": None, "2": None, "3": None, "4": None, "5": None, "more than 5": None},
	},
	"car": {
	"type": "noul",
	"instructions": "Is there a car in this image?",
	},
	},
	)
	print(json.dumps(result["answers"], indent=1))
	```

	Output (probabilities rounded to three decimals):

	```json
	{
	"scene": {"type": "choice", "choice": "airport",
	"probabilities": {"airport": 0.966, "harbor": 0.000, "parking lot": 0.033, "residential area": 0.000, "farmland": 0.000},
	"confidence": 0.905},
	"airplanes": {"type": "choice", "choice": "5",
	"probabilities": {"0": 0.000, "1": 0.001, "2": 0.002, "3": 0.046, "4": 0.440, "5": 0.454, "more than 5": 0.057},
	"confidence": 0.463},
	"car": {"type": "noul", "noul": 0.978}
	}
	```

	The scene and the car are answered right and with confidence. The image has four airplanes: EO-Jev's count is a near
	tie between 4 (0.440) and 5 (0.454), and its low confidence (0.46) marks the answer for review instead of hiding the
	mistake. Counting on unfamiliar imagery is one of the limitations below. For `noul` questions, the value is the
	probability of "yes".

	All questions about the same image or images are answered in one call.

	## Asking questions

	A request has a state (free text: what the image is, its sensor or resolution, any context), one or more
	images, and a dictionary of questions. Each question has a `type`:

	\| Type \| `criteria` \| Answer \|
	\|---\|---\|---\|
	\| `choice` \| a dict of options (up to 255); each value is an optional description (`None` is fine) \| `choice` (the top option), `probabilities` per option, `confidence` \|
	\| `noul` \| none (a yes/no question) \| `noul`: the probability of "yes" \|
	\| `score` \| an ordered list of 2–10 levels, e.g. `["no damage", "minor damage", "major damage", "destroyed"]` \| `score` (the expected level index) and `probabilities` per level \|

	Several images go into one request, for example a pre- and post-event pair or a time series. Say in the state which is
	which:

	```python
	result = model.system_one(
	state="Image 1 is the earlier (pre-disaster) image and Image 2 the later (post-disaster) image of the same area.",
	images=["pre.png", "post.png"],
	questions={
	"damage": {
	"type": "score",
	"instructions": "How damaged is the building in the centre?",
	"criteria": ["no damage", "minor damage", "major damage", "destroyed"],
	}
	},
	)
	```

	The model also works as a `transformers` pipeline:
	`pipeline("decision", model="mx297/EO-Jev", trust_remote_code=True)(state=..., questions=..., images=...)`.

	Tips
	- Keep each answer's wording short and distinct. The probabilities are over the options you give, so include an
	option such as "none of these" or "0" when it can be the answer.
	- The probabilities are calibrated at the checkpoint's temperature. A per-question threshold on the top probability
	decides which answers to accept automatically.
	- The code that runs the model (`d3_runtime.py` and the other `d3_*.py` files) is d3-flash's runtime, unchanged.

	## Training

	- Base model: d3-flash, with LoRA adapters on the text tower, the vision merger and the vision blocks (the vision
	blocks at 0.2× the learning rate), merged after training.
	- Data: decision questions built from public Earth-observation training splits: scene and land-use labels, object
	presence and counting, visual question answering, multi-temporal change, building damage and disaster type,
	geo-localisation. About 20% of every batch replays general decision data with d3-flash's own probabilities as targets,
	to keep its general skills. Every training image that duplicates or crops an evaluation image was removed.
	- Loss: KL divergence to the target distribution: one-hot, soft ordinal labels (SORD) for counts and damage
	levels, or d3-flash's own probabilities.
	- Compute: 20,000 steps of 16 questions on 4 NVIDIA A100 GPUs.

	Training and evaluation code: [github.com/mbzuai-oryx/EO-Jev](https://github.com/mbzuai-oryx/EO-Jev).

	## Limitations

	- Counting on unfamiliar imagery, robustness to the order of answer options, geometric and cross-view reasoning, and
	calibration far from the training data remain the hardest cases.
	- EO-Jev was trained and evaluated on optical (RGB) imagery; SAR and multispectral inputs were not part of training.
	- It answers only with the options you provide. It cannot say that none applies unless such an option is offered.
	- Validate it on your own data before relying on it for decisions with consequences.

	## License

	Apache-2.0 ([LICENSE](LICENSE)), as d3-flash, which is built on
	[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Apache-2.0); see [NOTICE](NOTICE). The datasets used for
	training keep their own licences. The example image is from the UC Merced Land Use dataset (USGS National Map imagery,
	public domain).

	## Citation

	```bibtex
	@article{abouelhadid2026eojev,
	title = {{EO-Jev}: Decision Model for {Earth} Observation},
	author = {Abouelhadid, Mohamed Atef and Abuhani, Diaa Addeen and
	Munir, Muhammad Akhtar and Khan, Fahad Shahbaz and Khan, Salman},
	journal = {arXiv preprint arXiv:2611.03371},
	year = {2026}
	}
	```

	Please also cite d3-flash:

	```bibtex
	@misc{d3_flash_2026,
	title = {{d3-flash}: A Multimodal Foundation Decision Model},
	author = {{vLLM Semantic Router Team}},
	year = {2026},
	howpublished = {\url{https://huggingface.co/vllm-sr/d3-flash}}
	}
	```