projects/VLMEvalKit_Thinkmorph_llm/README.md

VLMEvalKit for Thinkmorph

This repository provides evaluation support for the ThinkMorph model based on VLMEvalKit. And this repo also supports all the benchmarks evaluated in our paper, including: VSP, VisPuzzle, ChartQA, VStar, BLINK-J, MMVP, SAT, BLINK, and CV-Bench.

Installation

1. Clone the Repository

git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git
cd VLMEvalKit_Thinkmorph

2. Install Dependencies

pip install -r requirements.txt

3. Install ThinkMorph Dependencies

Make sure you have the necessary dependencies for ThinkMorph

For UniAR evaluation, use the separate ThinkMorph-Eval environment with Torch 2.7.1+cu128 and install OURS[train,decoder] from the surrounding Spatial-MMCoT workspace. The UniAR adapter adds only the canonical system protocol and preserves the framework-generated user prompt byte-for-byte.

Quick Start

1️⃣ Configure API Keys

export OPENAI_API_KEY="your_api_key_here"

2️⃣ Edit the model configuration in vlmeval/config.py:

thinkmorph_series = {
    "thinkmorph": partial(
        ThinkMorph, 
        model_path="ThinkMorph/ThinkMorph-7B", 
        think=True, # Enable thinking mode
        understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images
        temperature=0.3, 
        max_think_token_n=4096, 
        save_dir="path_to_your_imgs_dir" # Directory to save generated images
    ),
}

2️⃣ Choose the benchmark you wanna eval, and run.

An example script is shown in run_thinkmorph.sh. If you wanna try more benchmark, check VLMEvalKit Features

python run.py \
    --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \
    --model thinkmorph \
    --judge gpt-5 \
    --work-dir ./results

UniAR with DeepSeek judge

The registered UniAR models cover the base checkpoint, the rank-64 two-/four- epoch adapters, the BitCE weight-1/weight-3 ablations, and the absolute-IG two-epoch adapter. This fork preserves VLMEvalKit's original verifier and uses deepseek-v4-flash only when rule-based prefetch cannot extract an answer.

export DEEPSEEK_API_KEY="$(<API_KEY)"
LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \
  --data BLINK_Jigsaw \
  --model uniar-thinkmorph-v1-ig-abs-r64-val5-2epoch \
  --judge deepseek-v4-flash --reuse --work-dir <output-dir>

API_KEY is ignored by Git. Prefer DEEPSEEK_API_KEY in managed jobs; the UniAR runner may load the ignored file through DEEPSEEK_API_KEY_FILE.

The UniAR runner also sets UNIAR_ROLLOUT_DIR=<output>/rollouts. Each generated sample atomically saves prompt/assistant/full token IDs, observation and generated visual codes, grids, event metadata, decoded token text, and a COMPLETE marker. RGB decoding is intentionally deferred; the workbook's extra_records column links each prediction to its rollout directory.

Interactive retries set the stable namespace VLMEVAL_EVAL_ID=RUN, so date or live Git changes cannot redirect rank shards into a new prediction directory. VLMEVAL_DUMP_EVERY=1 commits the rank pickle after every generated sample; the upstream evaluator and verifier are unchanged. VLMEVAL_SCHEDULING=dynamic uses one distributed atomic counter: every free worker claims the next remaining query. Rank 0 first merges the existing workbook/PREV cache and all historical worker-count shards, then broadcasts one recovery snapshot so every worker indexes the same pending list. A single-worker retry also reads multi-worker history without distributed collectives. Workers persist their own cumulative dictionaries; they never rewrite dynamic results as static rank partitions. After the completion barrier, only rank 0 merges, checks full dataset coverage and publishes the workbook. Recovery and publication failures are propagated to every dynamic rank. Keep dynamic rank pickles and raw rollout artifacts for retry; only the legacy static path removes its completed rank files. Retained failure placeholders do not replace a successful retry, and the existing ignore_failed policy still applies. Model generation, response formatting, judge and scoring rules are unchanged.

CPU regression tests are in tests/test_dynamic_inference_regression.py and tests/test_model_construction_env.py; tests/dynamic_inference_distributed_smoke.py exercises the same function bodies with real Gloo processes and a fake model. It does not require GPUs or registry/model imports. Historical snapshots retain the old implementation: recover a failed job only through a new linked source snapshot while preserving its prediction namespace and complete rollout pairs.

To replace scores from the retired custom verifier without rerunning model inference, point the native LLM verifier at the existing raw workbook:

python scripts/rescore_with_llm_judge.py --dataset MMVP \
  --eval-file <model>_MMVP.xlsx --judge deepseek-v4-flash --nproc 24

This preserves the prediction workbook, replaces _acc.csv, removes stale exact_matching derivatives, and publishes DeepSeek result/provenance files in the same evaluation directory.