repro-bundle/README.md

# GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

This repository is the official code release for our ICML 2026 paper: [GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models]()

GePBench is a large-scale, highly customizable multimodal benchmark for evaluating and enhancing the fundamental geometric perception capabilities of Multimodal Large Language Models (MLLMs), covering core visual perception abilities such as spatial awareness and fine-grained visual perception. This repository contains:

- The full **data-generation pipeline** (structured rule generation → figure rendering → VQA question synthesis) for producing both the evaluation benchmark and large-scale training data.
- The **evaluation harness** supporting a wide range of open-source and API-based VLMs (LLaVA, InternVL, Qwen2-VL, MiniCPM-V, GLM-4V, mPLUG-Owl3, Llama-3.2-Vision, Mantis-Idefics2, G-LLaVA, Math-LLaVA, QVQ, and more).
- Training and downstream-benchmark code for **LLaVA-1.5-GeP**, our geometric-perception enhanced LLaVA variant.

## Install

Python 3.10+ is recommended.

### Core environment (data generation + benchmark evaluation)

```shell
pip install -r deploy/requirements.txt
```

This covers the data-generation pipeline (`data/`) and the benchmark evaluation harness (`eval/gepbench.py`) for the standard MLLMs implemented in `common/vllm/`.

A few model evaluators have additional model-specific dependencies that are intentionally **not** installed by default — install them only if you actually run the corresponding model:

- `qwen-vl-utils` — required by Qwen2-VL and QVQ (`common/vllm/Qwen2_VL_Instruct.py`, `common/vllm/QVQ.py`).
- `torchvision` — required by InternVL2_5 (`common/vllm/InternVL2_5.py`).
- `vllm` — required by the standalone vLLM-LLaVA wrapper in `common/inference.py`.
- `flash-attn` — soft-imported (via `importlib.util.find_spec`) by Qwen2-VL, QVQ, mPLUG-Owl3, MiniCPM-V, Math-LLaVA, Math-PUMA, Llama-3.2-Vision, InternVL2_5, G-LLaVA. Installation requires a matching CUDA + torch build; follow the [official guide](https://github.com/Dao-AILab/flash-attention).

### LLaVA-1.5-GeP training environment (optional)

The LLaVA-1.5 fork shipped under `llava/` requires `transformers==4.31.0`, which is **incompatible** with the modern MLLMs evaluated by `eval/gepbench.py` (Qwen2-VL, QVQ, InternVL2_5, Llama-3.2-Vision, ...). To train or finetune LLaVA-1.5-GeP, set up a **separate** virtual env and run:

```shell
pip install -r deploy/requirements-train.txt
```

See `docs/LLaVA-GeP.md` for the full LLaVA-1.5-GeP training and downstream-benchmark instructions.

## Running a module

A `./run` is provided for running a module. Examples:

```shell
chmod +x run

# run the file: default entry is main(); no argument is expected
./run -m data.rule.generate --num_basic_geo_samples 10

# you can also specify the entry function (--action); no argument is expected
./run -m data.format --action to_llava

# If you ensure that `if __name__ == '__main__': main()` is present in the file
# You can also run the file directly via python
python -m data.rule.generate --num_basic_geo_samples 10
```

# Benchmark Overview

The benchmark is in VQA format, covering 6 aspects (`existence`, `counting`, `size`, `location`, `reference`, `relation`) and categorized into `easy` and `hard` splits. An example of each category:

![](docs/assets/gepbench.png)

## Evaluation on Test Set

### Test Data

We provide a standard open test set on [huggingface](https://huggingface.co/datasets/starreeze/gepbench). Please download and unzip to `./dataset`. `tree dataset --filelimit 10` should result in this:

```
dataset
├── figures  [xxx entries exceeds filelimit, not opening dir]
├── vqa-easy
│   ├── counting.jsonl
│   ├── existence.jsonl
│   ├── location.jsonl
│   ├── reference.jsonl
│   ├── relation.jsonl
│   └── size.jsonl
└── vqa-hard
    ├── counting.jsonl
    ├── existence.jsonl
    ├── location.jsonl
    ├── reference.jsonl
    ├── relation.jsonl
    └── size.jsonl
```

### Supported Models

[TODO] add detailed instruction

Our officially supported models can be found in `./common/vllm`. Please download the corresponding checkpoints from huggingface and save them in `./models` with the same directory name as the python module (extension excluded). You can download them via

```shell
huggingface-cli download org_name/{model_name}-{model_size} --local-dir models/{model_name}-{model_size}
```

After obtaining all the checkpoints, start evaluation with

```shell
scripts/eval/bench.sh --eval_model {model_name}-{model_size} --eval_batchsize {batchsize}
```

The evaluation results will be saved in `results/{model_name}-{model_size}`.

### Evaluate on Custom Models

The easiest way is to create a file `common/vllm/model_name.py`, write a class `GenerateModel` inherited from `common/vllm/base.py: GenerateModelBase`, and implement the its `__init__` and `generate` method. Please read the base class [common/vllm/base.py](common/vllm/base.py) first and refer to the LLaVA-1.5 model [common/vllm/llava.py](common/vllm/llava.py) as an example. After that, your model become one of the supported models and can be used in the same way as the officially supported models.

Another option is to perform your own generation process before calculating the accuracy by referring to [eval/gepbench.py](eval/gepbench.py) which implements the evaluation process. Temperature should be set to 0.0 and do_sample should be set to False.

## Constructing Training Set

To construct the large-scale training set, just run the following command:

```shell
scripts/data/generate.sh train
```

Of course, you can also generate your own test set in the same way. The only difference is the number of samples.

The following is a description on the three phases and corresponding parameters for constructing the data. Please read it for customizing the data generation process.

### Structured textual description

Running the following command can generate rules for geometric shapes in `dataset/rules.json`:

```shell
scripts/data/rule-{easy/hard}.sh
```

You can control the generation process with the following arguments:

- min_num_shapes: the minimum number of shapes in each sample. Default is 2
- num_samples_per_num_shapes: a list for setting `num_samples` for each `num_shapes`. The number of samples with `num_shapes=min_num_shapes + i` is `num_samples_per_num_shapes[i]`.

there are some arguments for controling the numerical characteristics of geometric shapes:

- in_canvas_area_thres: the area threshold for shapes in the canvas, between 0 and 1. A value of 1 means the entire shape has to be fully contained within the canvas. Default is 0.8

- polygon_points_min_distance: the minimum distance between any two points of a polygon. Default is 0.01
- rectangle_ratio_thres: the aspect ratio constraints of rectangle, where the two items in the list represent the lower bound and upper bound respectively. Default is [1.5, 3.0]
- general_quadrilateral_angle_thres: the minimun angular difference between the angle of a general quadrilateral and π / 2
- general_triangle_angle_thres: the minimun angular difference between the angle of a general triangle and π / 3
- line_min/max_length: control the min/max length of line(segment). Default is 0.2/0.5
- ellipse_ratio_thres: the aspect ratio constraints of ellipse, where the two items in the list represent the lower bound and upper bound respectively. Default is [1.5, 3.0]

and there are arguments for controling the proportion of different shapes and relations, for example:

- polygon_shape_level: the proportion of polygon in all shapes
- line_shape_level: the proportion of line in all shapes
- ...
- polygon_tangent_line_level: the proportion of generating a tangent line in all polygon relations
- polygon_shared_edge_level: the proportion of generating a new polygon that have a shared edge with a given polygon
- ellipse_concentric_level: the proportion of generating a set of ellipses that is concentric with a given ellipse
- ...

Each 'level' argument is an integer (with a default value) representing the relative level within its shape/relation block. For more details, please refer to `RuleArgs` in `common/args.py`. All 'level' arguments will be transformed into probabilities using L1 normalization (sum normalization).

For example, if more ellipse is expected, you can set a higher level for ellipse_shape_level:

```shell
./run -m data.rule.generate --polygon_shape_level 1 --line_shape_level 1 --ellipse_shape_level 3 --spiral_shape_level 1
```

For the output, each data sample contains two parts:

- **shapes**: parameters and special information of each geometric shape.
- **relations**: relationship between two shapes in form of `[head_shape_idx, tail_shape_idx, relation_type]`

Example output format:

```json
{
  "shapes": [
    {
      "type": "line"
      //...
    },
    {
      "type": "ellipse"
      //...
    }
  ],
  "relations": [[0, 1, "tangent line"]]
}
```

### Figure Rendering

To simply generate figures with default settings, use the following command:

```shell
scripts/data/draw-{easy/hard}.sh
```

<!-- Two python files, `pil_backend.py` and `plt_backend.py` is provided, in which the former one is written in pillow, providing continuous change of shape, and a relatively less noisy image; the latter, in comparison, provides hand-drawing line style and more natural noise. `plt_backend.py` is recommended to use and `draw.py` will automatically use this version. You can change the preferred version by setting argument `backend` to `plt` or `pil`. -->

<!-- To use `plt_backend.py`, the following arguments are expected: -->

Or you may want to look at the following arguments:

- rules: "list[dict[str, Any]]". Mandatory. The rules you would like to draw.
- random_seed: int|None. The default value is None. Control the random seed.
- randomize: bool. The default value is True. Enable the noise-applying procedure.
- size: "tuple[float, float]". The deault value is (6.4, 6.4).
- dpi: int. The default value is 100. dpi \* size = resolution.
- line_weight: int. The default value is 4. Control the line weight. If `randomize` is enabled, the line weight will be randomly chosen in a certain range near the value.
- line_style: str. The default value is "none". Control the line style, which can be "none", "xkcd", or "gradient". "None" will make line a normal line; "xkcd" will make line a hand-drawn line; "gradient" will make line a gradient line. Notice that `line_style` could be overridden by `randomize == False` if `line_style == "xkcd"`. In this case, the line style will be set to "none". Also note that `line_style == "xkcd"` will affect all shapes whilst `"gradient"` will affect only straight lines.
- color:None|tuple[int,int,int]. The default value is None. If a color in RGB form is provided, that rule will be drawn in the given color. The the value is None, that rule will be drawn in random colors.
- n_white_line:None|int. The default value is None. If an integer is given, the white lines will be drawn in that certain amount. Otherwise, the value is randomly chosen.
- white_line_range:float. The default value is 0.25. Indicate the maximum length of a white line.
- Gaussian_mean: float. The default value is 0. Control the mean value of the Gaussian noise. The higher the value is, the grayer the image will be.
- Gaussian_var: float. The default value is 10. Control the variance of the Gaussian Noise. The higher the value is, the stronger the Gaussian Noise will be.
- Gaussian_proba: float. The default value is 1. Control the probability of applying Gaussian noise.
- Perlin_lattice: int. The default value is 20. Control the number of lattices while generating Perlin noise. The value is not recommended to change and may cause the crash the the module.
- Perlin_power: float. The default value is 16. Control the power of the Perlin noise, will affect the contrast ratio of the noise and the image.
- Perlin_bias: float. The default value is -16. Control the bias of the Perlin noise. The lower it is, the brighter the image will be.
- Perlin_proba: float. The default value is 1. Control the probability of applying Perlin noise per shape.
- stylish: bool. The default value is False. Setting to true will sharpen the image.
- stylish_depth: int. The default value is 10. Control the depth of the sharpening.
- stylish_height: float. The default value is 3.1416 / 2.2. Control the height of the sharpening.
- stylish_alpha: float. The default value is 3.1416 / 4. Control the alpha of the sharpening.
- inline_noise: bool. The default value is True. Setting to true will apply noise to the line/ray/segment. It may not very obvious in default setting in which Perlin_power is relatively low and the line color is close to black.

### QA Generation

```shell
scripts/data/vqa.sh {easy/hard}
```

The questions will be generated (by default) in `data/vqa`.