Instructions to use AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4

SGLang

How to use AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen3-VL-2B-Instruct-GPTQ-Int4
```

Qwen3-VL-2B-Instruct-GPTQ-Int4

Commit History

Update README.md

87bba71
verified

qqc1989 commited on Apr 22

set devices: {0}

5b511cd
verified

wangli commited on Mar 9

update project

0c25383
verified

wangli commited on Mar 6

Update README.md

aad163c
verified

qqc1989 commited on Jan 21

Update requirements.txt

f4d2f50
verified

qqc1989 commited on Jan 15

Update README.md

5861779
verified

qqc1989 commited on Jan 15

Update requirements.txt

a7efa9d
verified

qqc1989 commited on Jan 15

update

bdb6853

lihongjie commited on Dec 30, 2025

remove python tokenizer server

304a966

lihongjie commited on Dec 30, 2025

打开prefill优化

536be0d

lihongjie commited on Dec 23, 2025

fix mrope bug

f159af3

lihongjie commited on Dec 22, 2025

update

136991a

lihongjie commited on Dec 11, 2025

更新 tokenizer

c0bc963

lihongjie commited on Dec 11, 2025

更新tokenizer为内置tokenizer

5004753

lihongjie commited on Dec 11, 2025

tokenizer更新为内置c++ tokenizer

b726122

lihongjie commited on Dec 11, 2025

更新为内置tokenizer

7796708

lihongjie commited on Dec 11, 2025