Instructions to use JasperHaozhe/AgentGen-Bench-Evaluator with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JasperHaozhe/AgentGen-Bench-Evaluator with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="JasperHaozhe/AgentGen-Bench-Evaluator")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("JasperHaozhe/AgentGen-Bench-Evaluator")
model = AutoModelForMultimodalLM.from_pretrained("JasperHaozhe/AgentGen-Bench-Evaluator", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JasperHaozhe/AgentGen-Bench-Evaluator with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JasperHaozhe/AgentGen-Bench-Evaluator"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasperHaozhe/AgentGen-Bench-Evaluator",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/JasperHaozhe/AgentGen-Bench-Evaluator

SGLang

How to use JasperHaozhe/AgentGen-Bench-Evaluator with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JasperHaozhe/AgentGen-Bench-Evaluator" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasperHaozhe/AgentGen-Bench-Evaluator",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JasperHaozhe/AgentGen-Bench-Evaluator" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasperHaozhe/AgentGen-Bench-Evaluator",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use JasperHaozhe/AgentGen-Bench-Evaluator with Docker Model Runner:
```
docker model run hf.co/JasperHaozhe/AgentGen-Bench-Evaluator
```

AgentGen-Bench-Evaluator

AgentGen-Bench-Evaluator is the official evaluator for AgentGen-Bench, the benchmark introduced by Search Beyond What Can Be Taught: A Co-Evolution Framework for Search-Enhanced Image Generation. AgentGen-Bench was previously named SearchGen-Bench; the new name reflects its broader use for evaluating agentic generation systems.

The evaluator is trained on top of Qwen3.5-9B. We recommend this released checkpoint for full benchmark reproducibility.

Serving with vLLM

Install a recent vLLM release, download serve_evaluator.sh, and run:

pip install vllm
bash serve_evaluator.sh

The endpoint is OpenAI-compatible and serves the model as AgentGen-Bench-Evaluator on port 7070.

Two serving details are required:

Enforce --chat-template-content-format openai. The evaluator consumes multimodal OpenAI-style message content; relying on automatic format detection can produce incompatible requests.
For multiple GPUs, use vLLM data parallelism. For example, bash serve_evaluator.sh --gpus 0,1,2,3 starts four data-parallel replicas with tensor parallel size one. The script derives the data-parallel size from the GPU list; --dp and --tp remain available for custom layouts.

An equivalent direct command is:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve JasperHaozhe/AgentGen-Bench-Evaluator \
  --served-model-name AgentGen-Bench-Evaluator \
  --chat-template-content-format openai \
  --max-model-len 20480 \
  --limit-mm-per-prompt '{"image": 8, "video": 0}' \
  --tensor-parallel-size 1 \
  --data-parallel-size 4 \
  --gpu-memory-utilization 0.75 \
  --port 7070

Use the endpoint with the evaluation code in the SearchGen repository.

Intended use

This checkpoint is intended to reproduce AgentGen-Bench evaluation of generated images across knowledge and rendering dimensions. Like any learned judge, it may inherit biases from its training data and base model. Scores should be interpreted using the released protocol and aggregated across the benchmark, not as definitive judgments of individual images.

License

Apache-2.0. Users must also comply with the terms of the Qwen3.5-9B base model.

Downloads last month: 42

Safetensors

Model size

9B params

Tensor type

BF16

Model tree for JasperHaozhe/AgentGen-Bench-Evaluator

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Finetuned

(584)

this model