Buckets:

EmergentTwilight
/

MMCoT

Files

xet

EmergentTwilight/MMCoT / projects /VLMEvalKit_Thinkmorph_llm /README.md

EmergentTwilight

3 days ago

preview code

download

raw

5.56 kB

# VLMEvalKit for Thinkmorph

This repository provides evaluation support for the [ThinkMorph](https://arxiv.org/pdf/2510.27492) model based on [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). And this repo also supports all the benchmarks evaluated in our paper, including: `VSP`, `VisPuzzle`, `ChartQA`, `VStar`, `BLINK-J`, `MMVP`, `SAT`, `BLINK`, and `CV-Bench`.


## Installation

### 1. Clone the Repository

```
git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git
cd VLMEvalKit_Thinkmorph
```

### 2. Install Dependencies

```
pip install -r requirements.txt
```

### 3. Install ThinkMorph Dependencies

Make sure you have the necessary dependencies for ThinkMorph

For UniAR evaluation, use the separate `ThinkMorph-Eval` environment with
Torch 2.7.1+cu128 and install `OURS[train,decoder]` from the surrounding
Spatial-MMCoT workspace. The UniAR adapter adds only the canonical system
protocol and preserves the framework-generated user prompt byte-for-byte.


## Quick Start

1️⃣ Configure API Keys

```
export OPENAI_API_KEY="your_api_key_here"
```

2️⃣ Edit the model configuration in [`vlmeval/config.py`](vlmeval/config.py):
```
thinkmorph_series = {
    "thinkmorph": partial(
        ThinkMorph, 
        model_path="ThinkMorph/ThinkMorph-7B", 
        think=True, # Enable thinking mode
        understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images
        temperature=0.3, 
        max_think_token_n=4096, 
        save_dir="path_to_your_imgs_dir" # Directory to save generated images
    ),
}
```

2️⃣ Choose the benchmark you wanna eval, and run.

An example script is shown in [run_thinkmorph.sh](run_thinkmorph.sh). 
If  you wanna try more benchmark, check [VLMEvalKit Features](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb&view=vewUH3kRGs)

```
python run.py \
    --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \
    --model thinkmorph \
    --judge gpt-5 \
    --work-dir ./results
```

### UniAR with DeepSeek judge

The registered UniAR models cover the base checkpoint, the rank-64 two-/four-
epoch adapters, the BitCE weight-1/weight-3 ablations, and the absolute-IG
two-epoch adapter. This fork preserves VLMEvalKit's original verifier and uses
`deepseek-v4-flash` only when rule-based prefetch cannot extract an answer.

```bash
export DEEPSEEK_API_KEY="$(<API_KEY)"
LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \
  --data BLINK_Jigsaw \
  --model uniar-thinkmorph-v1-ig-abs-r64-val5-2epoch \
  --judge deepseek-v4-flash --reuse --work-dir <output-dir>
```

`API_KEY` is ignored by Git. Prefer `DEEPSEEK_API_KEY` in managed jobs; the
UniAR runner may load the ignored file through `DEEPSEEK_API_KEY_FILE`.

The UniAR runner also sets `UNIAR_ROLLOUT_DIR=<output>/rollouts`. Each generated
sample atomically saves prompt/assistant/full token IDs, observation and
generated visual codes, grids, event metadata, decoded token text, and a
`COMPLETE` marker. RGB decoding is intentionally deferred; the workbook's
`extra_records` column links each prediction to its rollout directory.

Interactive retries set the stable namespace `VLMEVAL_EVAL_ID=RUN`, so date or
live Git changes cannot redirect rank shards into a new prediction directory.
`VLMEVAL_DUMP_EVERY=1` commits the rank pickle after every generated sample;
the upstream evaluator and verifier are unchanged.
`VLMEVAL_SCHEDULING=dynamic` uses one distributed atomic counter: every free
worker claims the next remaining query. Rank 0 first merges the existing
workbook/PREV cache and all historical worker-count shards, then broadcasts one
recovery snapshot so every worker indexes the same pending list. A single-worker
retry also reads multi-worker history without distributed collectives.
Workers persist their own cumulative dictionaries; they never rewrite dynamic
results as static rank partitions. After the completion barrier, only rank 0
merges, checks full dataset coverage and publishes the workbook. Recovery and
publication failures are propagated to every dynamic rank. Keep dynamic rank
pickles and raw rollout artifacts for retry; only the legacy static path removes
its completed rank files. Retained failure placeholders do not replace a
successful retry, and the existing `ignore_failed` policy still applies.
Model generation, response formatting, judge and scoring rules are unchanged.

CPU regression tests are in `tests/test_dynamic_inference_regression.py` and
`tests/test_model_construction_env.py`; `tests/dynamic_inference_distributed_smoke.py`
exercises the same function bodies with real Gloo processes and a fake model.
It does not require GPUs or registry/model imports. Historical snapshots retain
the old implementation: recover a failed job only through a new linked source
snapshot while preserving its prediction namespace and complete rollout pairs.

To replace scores from the retired custom verifier without rerunning model
inference, point the native LLM verifier at the existing raw workbook:

```bash
python scripts/rescore_with_llm_judge.py --dataset MMVP \
  --eval-file <model>_MMVP.xlsx --judge deepseek-v4-flash --nproc 24
```

This preserves the prediction workbook, replaces `_acc.csv`, removes stale
`exact_matching` derivatives, and publishes DeepSeek result/provenance files
in the same evaluation directory.

Xet Storage Details

Size:: 5.56 kB
Xet hash:: f81fa4295217bdca78548309005e510306e8095ea9237abd0fade0a740efc1fa

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# VLMEvalKit for Thinkmorph

	This repository provides evaluation support for the [ThinkMorph](https://arxiv.org/pdf/2510.27492) model based on [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). And this repo also supports all the benchmarks evaluated in our paper, including: `VSP`, `VisPuzzle`, `ChartQA`, `VStar`, `BLINK-J`, `MMVP`, `SAT`, `BLINK`, and `CV-Bench`.


	## Installation

	### 1. Clone the Repository

	```
	git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git
	cd VLMEvalKit_Thinkmorph
	```

	### 2. Install Dependencies

	```
	pip install -r requirements.txt
	```

	### 3. Install ThinkMorph Dependencies

	Make sure you have the necessary dependencies for ThinkMorph

	For UniAR evaluation, use the separate `ThinkMorph-Eval` environment with
	Torch 2.7.1+cu128 and install `OURS[train,decoder]` from the surrounding
	Spatial-MMCoT workspace. The UniAR adapter adds only the canonical system
	protocol and preserves the framework-generated user prompt byte-for-byte.


	## Quick Start

	1️⃣ Configure API Keys

	```
	export OPENAI_API_KEY="your_api_key_here"
	```

	2️⃣ Edit the model configuration in [`vlmeval/config.py`](vlmeval/config.py):
	```
	thinkmorph_series = {
	"thinkmorph": partial(
	ThinkMorph,
	model_path="ThinkMorph/ThinkMorph-7B",
	think=True, # Enable thinking mode
	understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images
	temperature=0.3,
	max_think_token_n=4096,
	save_dir="path_to_your_imgs_dir" # Directory to save generated images
	),
	}
	```

	2️⃣ Choose the benchmark you wanna eval, and run.

	An example script is shown in [run_thinkmorph.sh](run_thinkmorph.sh).
	If you wanna try more benchmark, check [VLMEvalKit Features](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb&view=vewUH3kRGs)

	```
	python run.py \
	--data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \
	--model thinkmorph \
	--judge gpt-5 \
	--work-dir ./results
	```

	### UniAR with DeepSeek judge

	The registered UniAR models cover the base checkpoint, the rank-64 two-/four-
	epoch adapters, the BitCE weight-1/weight-3 ablations, and the absolute-IG
	two-epoch adapter. This fork preserves VLMEvalKit's original verifier and uses
	`deepseek-v4-flash` only when rule-based prefetch cannot extract an answer.

	```bash
	export DEEPSEEK_API_KEY="$(<API_KEY)"
	LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
	SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \
	--data BLINK_Jigsaw \
	--model uniar-thinkmorph-v1-ig-abs-r64-val5-2epoch \
	--judge deepseek-v4-flash --reuse --work-dir <output-dir>
	```

	`API_KEY` is ignored by Git. Prefer `DEEPSEEK_API_KEY` in managed jobs; the
	UniAR runner may load the ignored file through `DEEPSEEK_API_KEY_FILE`.

	The UniAR runner also sets `UNIAR_ROLLOUT_DIR=<output>/rollouts`. Each generated
	sample atomically saves prompt/assistant/full token IDs, observation and
	generated visual codes, grids, event metadata, decoded token text, and a
	`COMPLETE` marker. RGB decoding is intentionally deferred; the workbook's
	`extra_records` column links each prediction to its rollout directory.

	Interactive retries set the stable namespace `VLMEVAL_EVAL_ID=RUN`, so date or
	live Git changes cannot redirect rank shards into a new prediction directory.
	`VLMEVAL_DUMP_EVERY=1` commits the rank pickle after every generated sample;
	the upstream evaluator and verifier are unchanged.
	`VLMEVAL_SCHEDULING=dynamic` uses one distributed atomic counter: every free
	worker claims the next remaining query. Rank 0 first merges the existing
	workbook/PREV cache and all historical worker-count shards, then broadcasts one
	recovery snapshot so every worker indexes the same pending list. A single-worker
	retry also reads multi-worker history without distributed collectives.
	Workers persist their own cumulative dictionaries; they never rewrite dynamic
	results as static rank partitions. After the completion barrier, only rank 0
	merges, checks full dataset coverage and publishes the workbook. Recovery and
	publication failures are propagated to every dynamic rank. Keep dynamic rank
	pickles and raw rollout artifacts for retry; only the legacy static path removes
	its completed rank files. Retained failure placeholders do not replace a
	successful retry, and the existing `ignore_failed` policy still applies.
	Model generation, response formatting, judge and scoring rules are unchanged.

	CPU regression tests are in `tests/test_dynamic_inference_regression.py` and
	`tests/test_model_construction_env.py`; `tests/dynamic_inference_distributed_smoke.py`
	exercises the same function bodies with real Gloo processes and a fake model.
	It does not require GPUs or registry/model imports. Historical snapshots retain
	the old implementation: recover a failed job only through a new linked source
	snapshot while preserving its prediction namespace and complete rollout pairs.

	To replace scores from the retired custom verifier without rerunning model
	inference, point the native LLM verifier at the existing raw workbook:

	```bash
	python scripts/rescore_with_llm_judge.py --dataset MMVP \
	--eval-file <model>_MMVP.xlsx --judge deepseek-v4-flash --nproc 24
	```

	This preserves the prediction workbook, replaces `_acc.csv`, removes stale
	`exact_matching` derivatives, and publishes DeepSeek result/provenance files
	in the same evaluation directory.