Instructions to use AXERA-TECH/InternVL3_5-1B_GPTQ_INT4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/InternVL3_5-1B_GPTQ_INT4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AXERA-TECH/InternVL3_5-1B_GPTQ_INT4")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/InternVL3_5-1B_GPTQ_INT4", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/InternVL3_5-1B_GPTQ_INT4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/InternVL3_5-1B_GPTQ_INT4

SGLang

How to use AXERA-TECH/InternVL3_5-1B_GPTQ_INT4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/InternVL3_5-1B_GPTQ_INT4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/InternVL3_5-1B_GPTQ_INT4 with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/InternVL3_5-1B_GPTQ_INT4
```

InternVL3_5-1B_GPTQ_INT4

Commit History

update readme

d02db73

yongqiang commited on Jan 26

update readme

2d49224

yongqiang commited on Jan 26

update readme

c5991d4

yongqiang commited on Jan 26

add ax650_api exe script & update readme

f830cf1

yongqiang commited on Jan 26

upload ax620e(630c) execution script

d838478

yongqiang commited on Jan 26

upload ax620e(630c) models

873aef6

yongqiang commited on Jan 26

add internvl3-5 c++ demo with Xet storage

138454d

yongqiang commited on Jan 9

perf(ttft): improve model response speed

964caa9

yongqiang commited on Jan 7

兼容 gptq config

6a33ea5

yongqiang commited on Dec 26, 2025

Update README

ffa96d0

yongqiang commited on Dec 26, 2025

Initialize the repository

c5c0a20

yongqiang commited on Dec 26, 2025

Commit History

update readme d02db73

update readme 2d49224

update readme c5991d4

add ax650_api exe script & update readme f830cf1

upload ax620e(630c) execution script d838478

upload ax620e(630c) models 873aef6

add internvl3-5 c++ demo with Xet storage 138454d

perf(ttft): improve model response speed 964caa9

兼容 gptq config 6a33ea5

Update README ffa96d0

Initialize the repository c5c0a20

update readme

d02db73

update readme

2d49224

update readme

c5991d4

add ax650_api exe script & update readme

f830cf1

upload ax620e(630c) execution script

d838478

upload ax620e(630c) models

873aef6

add internvl3-5 c++ demo with Xet storage

138454d

perf(ttft): improve model response speed

964caa9

兼容 gptq config

6a33ea5

Update README

ffa96d0

Initialize the repository

c5c0a20