junwatu/repro-gepbench-artifacts/repro-bundle - Storage Bucket

GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

This repository is the official code release for our ICML 2026 paper: GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models

GePBench is a large-scale, highly customizable multimodal benchmark for evaluating and enhancing the fundamental geometric perception capabilities of Multimodal Large Language Models (MLLMs), covering core visual perception abilities such as spatial awareness and fine-grained visual perception. This repository contains:

• The full data-generation pipeline (structured rule generation → figure rendering → VQA question synthesis) for producing both the evaluation benchmark and large-scale training data.
• The evaluation harness supporting a wide range of open-source and API-based VLMs (LLaVA, InternVL, Qwen2-VL, MiniCPM-V, GLM-4V, mPLUG-Owl3, Llama-3.2-Vision, Mantis-Idefics2, G-LLaVA, Math-LLaVA, QVQ, and more).
• Training and downstream-benchmark code for LLaVA-1.5-GeP, our geometric-perception enhanced LLaVA variant.

Install

Python 3.10+ is recommended.

Core environment (data generation + benchmark evaluation)

pip install -r deploy/requirements.txt

This covers the data-generation pipeline (data/) and the benchmark evaluation harness (eval/gepbench.py) for the standard MLLMs implemented in common/vllm/.

A few model evaluators have additional model-specific dependencies that are intentionally not installed by default — install them only if you actually run the corresponding model:

• qwen-vl-utils — required by Qwen2-VL and QVQ (common/vllm/Qwen2_VL_Instruct.py, common/vllm/QVQ.py).
• torchvision — required by InternVL2_5 (common/vllm/InternVL2_5.py).
• vllm — required by the standalone vLLM-LLaVA wrapper in common/inference.py.
• flash-attn — soft-imported (via importlib.util.find_spec) by Qwen2-VL, QVQ, mPLUG-Owl3, MiniCPM-V, Math-LLaVA, Math-PUMA, Llama-3.2-Vision, InternVL2_5, G-LLaVA. Installation requires a matching CUDA + torch build; follow the official guide.

LLaVA-1.5-GeP training environment (optional)

The LLaVA-1.5 fork shipped under llava/ requires transformers==4.31.0, which is incompatible with the modern MLLMs evaluated by eval/gepbench.py (Qwen2-VL, QVQ, InternVL2_5, Llama-3.2-Vision, ...). To train or finetune LLaVA-1.5-GeP, set up a separate virtual env and run:

pip install -r deploy/requirements-train.txt

See docs/LLaVA-GeP.md for the full LLaVA-1.5-GeP training and downstream-benchmark instructions.

Running a module

A ./run is provided for running a module. Examples:

chmod +x run

# run the file: default entry is main(); no argument is expected
./run -m data.rule.generate --num_basic_geo_samples 10

# you can also specify the entry function (--action); no argument is expected
./run -m data.format --action to_llava

# If you ensure that `if __name__ == '__main__': main()` is present in the file
# You can also run the file directly via python
python -m data.rule.generate --num_basic_geo_samples 10

Benchmark Overview

The benchmark is in VQA format, covering 6 aspects (existence, counting, size, location, reference, relation) and categorized into easy and hard splits. An example of each category:

Evaluation on Test Set

Test Data

We provide a standard open test set on huggingface. Please download and unzip to ./dataset. tree dataset --filelimit 10 should result in this:

dataset
├── figures  [xxx entries exceeds filelimit, not opening dir]
├── vqa-easy
│   ├── counting.jsonl
│   ├── existence.jsonl
│   ├── location.jsonl
│   ├── reference.jsonl
│   ├── relation.jsonl
│   └── size.jsonl
└── vqa-hard
    ├── counting.jsonl
    ├── existence.jsonl
    ├── location.jsonl
    ├── reference.jsonl
    ├── relation.jsonl
    └── size.jsonl

Supported Models

[TODO] add detailed instruction

Our officially supported models can be found in ./common/vllm. Please download the corresponding checkpoints from huggingface and save them in ./models with the same directory name as the python module (extension excluded). You can download them via

huggingface-cli download org_name/{model_name}-{model_size} --local-dir models/{model_name}-{model_size}

After obtaining all the checkpoints, start evaluation with

scripts/eval/bench.sh --eval_model {model_name}-{model_size} --eval_batchsize {batchsize}

The evaluation results will be saved in results/{model_name}-{model_size}.

Evaluate on Custom Models

The easiest way is to create a file common/vllm/model_name.py, write a class GenerateModel inherited from common/vllm/base.py: GenerateModelBase, and implement the its __init__ and generate method. Please read the base class common/vllm/base.py first and refer to the LLaVA-1.5 model common/vllm/llava.py as an example. After that, your model become one of the supported models and can be used in the same way as the officially supported models.

Another option is to perform your own generation process before calculating the accuracy by referring to eval/gepbench.py which implements the evaluation process. Temperature should be set to 0.0 and do_sample should be set to False.

Constructing Training Set

To construct the large-scale training set, just run the following command:

scripts/data/generate.sh train

Of course, you can also generate your own test set in the same way. The only difference is the number of samples.

The following is a description on the three phases and corresponding parameters for constructing the data. Please read it for customizing the data generation process.

Structured textual description

Running the following command can generate rules for geometric shapes in dataset/rules.json:

scripts/data/rule-{easy/hard}.sh

You can control the generation process with the following arguments:

• min_num_shapes: the minimum number of shapes in each sample. Default is 2
• num_samples_per_num_shapes: a list for setting num_samples for each num_shapes. The number of samples with num_shapes=min_num_shapes + i is num_samples_per_num_shapes[i].

there are some arguments for controling the numerical characteristics of geometric shapes:

• in_canvas_area_thres: the area threshold for shapes in the canvas, between 0 and 1. A value of 1 means the entire shape has to be fully contained within the canvas. Default is 0.8

• polygon_points_min_distance: the minimum distance between any two points of a polygon. Default is 0.01

• rectangle_ratio_thres: the aspect ratio constraints of rectangle, where the two items in the list represent the lower bound and upper bound respectively. Default is [1.5, 3.0]

• general_quadrilateral_angle_thres: the minimun angular difference between the angle of a general quadrilateral and π / 2

• general_triangle_angle_thres: the minimun angular difference between the angle of a general triangle and π / 3

• line_min/max_length: control the min/max length of line(segment). Default is 0.2/0.5

• ellipse_ratio_thres: the aspect ratio constraints of ellipse, where the two items in the list represent the lower bound and upper bound respectively. Default is [1.5, 3.0]

and there are arguments for controling the proportion of different shapes and relations, for example:

• polygon_shape_level: the proportion of polygon in all shapes
• line_shape_level: the proportion of line in all shapes
• ...
• polygon_tangent_line_level: the proportion of generating a tangent line in all polygon relations
• polygon_shared_edge_level: the proportion of generating a new polygon that have a shared edge with a given polygon
• ellipse_concentric_level: the proportion of generating a set of ellipses that is concentric with a given ellipse
• ...

Each 'level' argument is an integer (with a default value) representing the relative level within its shape/relation block. For more details, please refer to RuleArgs in common/args.py. All 'level' arguments will be transformed into probabilities using L1 normalization (sum normalization).

For example, if more ellipse is expected, you can set a higher level for ellipse_shape_level:

./run -m data.rule.generate --polygon_shape_level 1 --line_shape_level 1 --ellipse_shape_level 3 --spiral_shape_level 1

For the output, each data sample contains two parts:

• shapes: parameters and special information of each geometric shape.
• relations: relationship between two shapes in form of [head_shape_idx, tail_shape_idx, relation_type]

Example output format:

{
  "shapes": [
    {
      "type": "line"
      //...
    },
    {
      "type": "ellipse"
      //...
    }
  ],
  "relations": [[0, 1, "tangent line"]]
}

Figure Rendering

To simply generate figures with default settings, use the following command:

scripts/data/draw-{easy/hard}.sh

Or you may want to look at the following arguments:

• rules: "list[dict[str, Any]]". Mandatory. The rules you would like to draw.
• random_seed: int|None. The default value is None. Control the random seed.
• randomize: bool. The default value is True. Enable the noise-applying procedure.
• size: "tuple[float, float]". The deault value is (6.4, 6.4).
• dpi: int. The default value is 100. dpi * size = resolution.
• line_weight: int. The default value is 4. Control the line weight. If randomize is enabled, the line weight will be randomly chosen in a certain range near the value.
• line_style: str. The default value is "none". Control the line style, which can be "none", "xkcd", or "gradient". "None" will make line a normal line; "xkcd" will make line a hand-drawn line; "gradient" will make line a gradient line. Notice that line_style could be overridden by randomize == False if line_style == "xkcd". In this case, the line style will be set to "none". Also note that line_style == "xkcd" will affect all shapes whilst "gradient" will affect only straight lines.
• color:None|tuple[int,int,int]. The default value is None. If a color in RGB form is provided, that rule will be drawn in the given color. The the value is None, that rule will be drawn in random colors.
• n_white_line:None|int. The default value is None. If an integer is given, the white lines will be drawn in that certain amount. Otherwise, the value is randomly chosen.
• white_line_range:float. The default value is 0.25. Indicate the maximum length of a white line.
• Gaussian_mean: float. The default value is 0. Control the mean value of the Gaussian noise. The higher the value is, the grayer the image will be.
• Gaussian_var: float. The default value is 10. Control the variance of the Gaussian Noise. The higher the value is, the stronger the Gaussian Noise will be.
• Gaussian_proba: float. The default value is 1. Control the probability of applying Gaussian noise.
• Perlin_lattice: int. The default value is 20. Control the number of lattices while generating Perlin noise. The value is not recommended to change and may cause the crash the the module.
• Perlin_power: float. The default value is 16. Control the power of the Perlin noise, will affect the contrast ratio of the noise and the image.
• Perlin_bias: float. The default value is -16. Control the bias of the Perlin noise. The lower it is, the brighter the image will be.
• Perlin_proba: float. The default value is 1. Control the probability of applying Perlin noise per shape.
• stylish: bool. The default value is False. Setting to true will sharpen the image.
• stylish_depth: int. The default value is 10. Control the depth of the sharpening.
• stylish_height: float. The default value is 3.1416 / 2.2. Control the height of the sharpening.
• stylish_alpha: float. The default value is 3.1416 / 4. Control the alpha of the sharpening.
• inline_noise: bool. The default value is True. Setting to true will apply noise to the line/ray/segment. It may not very obvious in default setting in which Perlin_power is relatively low and the line color is close to black.

QA Generation

scripts/data/vqa.sh {easy/hard}

The questions will be generated (by default) in data/vqa.