README.md

<div align="center">
  <h2><strong>Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing</strong></h2>
  <h5>
    <em>
      Fengxiang Wang<sup>1,*</sup>, Jiangnan Huang<sup>2,*</sup>, Mingshuo Chen<sup>1</sup>, Yueying Li<sup>1</sup>, <br/>
      Yang Shi<sup>1</sup>, Junwei Luo<sup>2</sup>, Haoyu Wang<sup>3,†</sup>, Yansheng Li<sup>2</sup>, Jing Zhang<sup>2</sup>, <br/>
      Haiyan Zhao<sup>3,†</sup>, Wenjing Yang<sup>1</sup>
    </em>
    <br/><br/>
    <sup>1</sup> National University of Defense Technology &nbsp;
    <sup>2</sup> Wuhan University &nbsp;
    <sup>3</sup> Tsinghua University
    <br/>
    <sup>*</sup> Equal contribution &nbsp; <sup>†</sup> Corresponding authors
  </h5>
  <p>
    📃 <a href="https://arxiv.org/abs/2607.25993" target="_blank">Paper</a> |
    🤗 <a href="https://huggingface.co/NaOHjiang/GeoLens" target="_blank">GeoLens Model</a> |
    🤗 <a href="https://huggingface.co/datasets/NaOHjiang/GeoMTVR" target="_blank">GeoMTVR Dataset</a> |
    🏆 <a href="https://github.com/AI9Stars/XLRS-Bench" target="_blank">XLRS-Bench</a>
  </p>
</div>


---

# 📚 Contents

- [📚 Contents](#-contents)
- [🔥News](#news)
- [📜Abstract](#abstract)
- [🔍Method](#method)
  - [Supervised Fine-Tuning (SFT)](#supervised-fine-tuning-sft)
  - [Tool-Attention-Focused Reinforcement Learning (RTAL)](#tool-attention-focused-reinforcement-learning-rtal)
- [🚀Reproduction](#reproduction)
  - [Environment](#environment)
  - [SFT](#sft)
  - [RTAL](#rtal)
  - [XLRS-Bench](#xlrs-bench)
  - [RSHR](#rshr)
  - [LRS-GRO-eval](#lrs-gro-eval)
- [📊Results](#results)
- [🔗Citation](#citation)
- [🤝Acknowledgement](#acknowledgement)

# 🔥News

* **[2026.07]** Paper released on [arXiv](https://arxiv.org/abs/2607.25993).
* **[2026.06]** Model weights and code released.
* **[2026.06]** GeoMTVR dataset released.

# 📜Abstract

Ultra-high-resolution (UHR) remote-sensing (RS) imagery provides fine-grained Earth-observation evidence over city-scale scenes, but poses a fundamental challenge for multimodal large language models (MLLMs): task-relevant evidence is often sparse, local, and spatially dispersed across extremely large visual contexts. A natural solution is to equip MLLMs with zoom-in tools for active local inspection. However, through a pilot study on XLRS-Bench, we find that zoom-in is only partially effective: it resolves easy and medium-level tasks with locally recoverable evidence, but saturates on hard cases requiring global search, multi-region comparison, path planning, or dispersed-evidence reasoning.

Motivated by this finding, we move beyond single-tool zoom-in and introduce **GeoMTVR**, a large-scale Geospatial Multi-Tool Visual Reasoning dataset built from wide-area satellite imagery. GeoMTVR contains **13K** UHR VQA samples with interleaved reasoning trajectories, diverse visual tool calls, and returned visual observations, enabling models to learn question decomposition, tool selection, regional inspection, object-level grounding, auxiliary visual reasoning, and cross-tool evidence integration.

Beyond supervised fine-tuning, we propose a **tool-attention-focused reinforcement learning** algorithm (RTAL) that concentrates optimization on critical tool-use decisions, including when to invoke tools, which tool to select, where to apply it, and how to interpret tool outputs.

By combining SFT on GeoMTVR with our RL algorithm, we develop **GeoLens**, a multi-tool visual reasoning MLLM for UHR RS. Experiments show that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, achieving stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.

## Model Weights

| Model | Description | Link |
|-------|-------------|------|
| **GeoLens** | Final multi-tool visual reasoning model (SFT + RTAL) | [🤗 NaOHjiang/GeoLens](https://huggingface.co/NaOHjiang/GeoLens) |
| **GeoMTVR** | SFT training dataset (13K UHR VQA samples with tool trajectories) | [🤗 NaOHjiang/GeoMTVR](https://huggingface.co/datasets/NaOHjiang/GeoMTVR) |

## Repository Structure

```
GeoLens/
  sft/                         # SFT configs and data-building scripts
  rtal/RTAL_dl_final/          # RTAL RL training code (dual JSD loss)
  evaluation/
    EVAL_by_myself/            # XLRS-Bench eval & score recomputation
    RSHR_eval/                 # RSHR benchmark evaluation
    LRSGRO_eval/               # LRS-GRO-eval evaluation
  scripts/                     # Reproduction wrappers (SFT, RTAL, eval)
  docs/                        # Source manifest and notes
```

# 🔍Method

## Supervised Fine-Tuning (SFT)

We fine-tune Qwen2.5-VL-7B-Instruct on the GeoMTVR dataset, which contains **13K** UHR VQA samples with interleaved tool-use trajectories. The SFT stage teaches the model to follow tool-use instructions, perform regional inspections, and integrate multi-tool evidence.

## Tool-Attention-Focused Reinforcement Learning (RTAL)

We propose **RTAL** (Reinforced Tool-Attention Learning), a novel RL algorithm that applies a dual JSD (Jensen-Shannon Divergence) loss with **key-mask** and **valid-mask** to concentrate optimization on critical tool-use decisions:

- **When to invoke tools**: The model learns to decide whether current visual evidence is sufficient or additional tool calls are needed.
- **Which tool to select**: The model learns to choose the appropriate tool (zoom-in, region inspection, object grounding, etc.) based on the task requirement.
- **Where to apply it**: The model learns to localize the correct spatial region for tool application.
- **How to interpret tool outputs**: The model learns to integrate tool-returned visual observations into its reasoning chain.

# 🚀Reproduction

## Environment

```bash
git clone https://github.com/NaOH3021/GeoLens.git
cd GeoLens
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -e rtal/RTAL_dl_final/verl
```

## SFT

The SFT stage fine-tunes Qwen2.5-VL-7B-Instruct on the GeoMTVR dataset to obtain the SFT checkpoint.

### Step 1: Download Dependencies

| Dependency | Source | Notes |
|------------|--------|-------|
| **Qwen2.5-VL-7B-Instruct** | [🤗 Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) | Base model |
| **GeoMTVR dataset** | [🤗 NaOHjiang/GeoMTVR](https://huggingface.co/datasets/NaOHjiang/GeoMTVR) | Multi-tool visual reasoning dataset (13K samples) |
| **LLaMA-Factory** | `pip install llamafactory` | Training framework used by the SFT script |

Place the base model at a local path, e.g., `/path/to/models/Qwen2.5-VL-7B-Instruct`.

### Step 2: Download and Prepare Dataset

Download the GeoMTVR dataset from [HuggingFace](https://huggingface.co/datasets/NaOHjiang/GeoMTVR). The repository contains annotation files (`dataset_info.json` + 3 JSONL files) and image chunks:

```bash
# Download the dataset repository
git lfs clone https://huggingface.co/datasets/NaOHjiang/GeoMTVR

# Or download files manually via the web interface (see "Files" tab)

# Merge and extract the image archive
cd GeoMTVR
cat images/geomtvr_images_chunk_*.tar.part > geomtvr_images.tar
tar xf geomtvr_images.tar
```

Organize the extracted data into a single dataset directory (LLaMA-Factory format):

```
/path/to/geomtvr-data/
  cold_start/
    super_accepted_or_route.3img_summary.jsonl
    zoom_results.visual.qa.with_aux.full.strict_superstyle.jsonl
    route_visual_accepted_388.superstyle_sft.jsonl
  images/
    aux_images/...
    vqa_results/...
  dataset_info.json
```

Refer to [LLaMA-Factory dataset preparation](https://github.com/hiyouga/LLaMA-Factory/blob/main/data/README.md) for the exact format.

### Step 3: Run SFT

```bash
BASE_MODEL=/path/to/models/Qwen2.5-VL-7B-Instruct \
DATASET_DIR=/path/to/geomtvr-data \
OUTPUT_DIR=/path/to/output/my_model_SFT_13078 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
bash scripts/run_sft.sh
```

The script automatically injects your paths into the config template at `sft/cold_start_qwen25vl_8gpu/config_cold_start_super_plus_lrs_8gpu_20260408.yaml`. If using a different number of GPUs or DeepSpeed config, modify the following in the template:

- `model_name_or_path` → your `BASE_MODEL` path
- `dataset_dir` → your `DATASET_DIR` path
- `output_dir` → your `OUTPUT_DIR` path
- `deepspeed` → path to your DeepSpeed config (ZeRO-3 recommended for full-parameter tuning)

> **Note**: We do not release the SFT checkpoint. The SFT step must be run locally before proceeding to RTAL. Training on 8×A100-80GB takes approximately 2 days.

## RTAL

RTAL (Reinforced Tool-Attention Learning) starts from the SFT checkpoint and uses key-mask + valid-mask dual JSD loss with rollout `n=8`. The training data consists of parquet files with the following schema: prompt (with tool-call format), response, and reward scores.

### Step 1: Prepare Data

Convert the GeoMTVR dataset into the parquet format expected by the [verl](https://github.com/volcengine/verl) framework:

```
/path/to/rtal_data/
  train.parquet          # Training set with tool-interleaved prompts and responses
  val.parquet            # Validation set
```

Each parquet row must contain at least `prompt` and `response` columns. Refer to the data bundle structure in `rtal/RTAL_dl_final/repro_supervqa_rl748/data_bundles/` for the exact schema.

### Step 2: Run RTAL

```bash
BASE_MODEL=/path/to/output/my_model_SFT_13078 \
TRAIN_PARQUET=/path/to/rtal_data/train.parquet \
VAL_PARQUET=/path/to/rtal_data/val.parquet \
OUTPUT_DIR=/path/to/rtal_outputs \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
PYTHON_BIN=/path/to/python \
bash scripts/run_rtal.sh
```

Key hyper-parameters (config: `rtal/RTAL_dl_final/repro_supervqa_rl748/config_supervqa_rl748_sft13078_rtal_dualloss_washed_super8066_lrsgro1913.yaml`):

- `attention_jsd_loss_coef_key_mask=0.01`
- `attention_jsd_loss_coef_valid_mask=0.01`
- `actor_rollout_ref.rollout.n=8`
- `max_prompt_length=8192`
- `max_response_length=1024`

> **Note**: If you prefer to skip SFT and directly run RTAL, you can use our released GeoLens checkpoint as a starting point: set `BASE_MODEL` to the downloaded GeoLens model path.

## Evaluation

### XLRS-Bench

```bash
DATASET_GLOB='/path/to/XLRS-Bench-lite_part*.jsonl' \
PYTHON_BIN=/path/to/python \
bash scripts/run_eval.sh /path/to/hf_model
```

Default generation uses:
```
max_new_tokens=8, temperature=0, do_sample=false, choice_constrained=true, choice_allow_space=true
```

Score recomputation:

```bash
bash scripts/recompute_xlrs_scores.sh \
  --samples_jsonl /path/to/*_samples_xlrs-lite-local-custom.jsonl \
  --results_json /path/to/*_results.json \
  --model_name MODEL_NAME \
  --summary_only
```

### RSHR

Evaluation code is in `evaluation/RSHR_eval/`. Download the benchmark images first:

```bash
cd evaluation/RSHR_eval
python download_xhrbench_images.py --save_dir /path/to/rshr_images
```

Run evaluation on a HuggingFace model:

```bash
python eval_qwen25vl_xhrbench.py \
  --model_path /path/to/hf_model \
  --save_dir /path/to/rshr_images \
  --output_dir results/
```

This evaluates MCQ, Open-Ended VQA, and Image Captioning tasks. See `evaluation/RSHR_eval/README.md` for details.

### LRS-GRO-eval

Evaluation code is in `evaluation/LRSGRO_eval/`. First download the ZoomEarth-3B monitor:

```bash
cd evaluation/LRSGRO_eval
bash download_zoomearth3b_with_monitor.sh
```

Generate inference results:

```bash
bash infer.sh /path/to/hf_model /path/to/output/inference.jsonl
```

Then compute accuracy:

```bash
bash eval.sh /path/to/output/inference.jsonl
```

See `evaluation/LRSGRO_eval/run_scripts/` for batch evaluation scripts on multiple models.

# 📊Results

GeoLens achieves state-of-the-art performance on three benchmarks:

| Benchmark | Metric | GeoLens |
|-----------|--------|---------|
| **XLRS-Bench** | Macro-13 Avg | **54.2%** |
| **GRO-eval** | Accuracy | **60.7%** |
| **RSHR** | Accuracy | **42.54%** |

Results demonstrate that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, with stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.

# 🔗Citation

If you find our work helpful, please consider citing:

```latex
@article{wang2026geolens,
  title={Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing},
  author={Fengxiang Wang and Jiangnan Huang and Mingshuo Chen and Yueying Li and Yang Shi and Junwei Luo and Haoyu Wang and Yansheng Li and Jing Zhang and Haiyan Zhao and Wenjing Yang},
  journal={arXiv preprint arXiv:2607.25993},
  year={2026}
}
```

# 🤝Acknowledgement

* [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL): The base model for GeoLens.
* [XLRS-Bench](https://github.com/AI9Stars/XLRS-Bench): Evaluation benchmark for ultra-high-resolution remote sensing.
* [verl](https://github.com/volcengine/verl): The RL training framework used for RTAL.
* [GeoLLaVA-8K](https://github.com/initiavcms/GeoLLaVA-8K): Prior work on UHR RS MLLMs.