Instructions to use OpenGVLab/InternVL2_5-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenGVLab/InternVL2_5-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="OpenGVLab/InternVL2_5-4B", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("OpenGVLab/InternVL2_5-4B", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use OpenGVLab/InternVL2_5-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenGVLab/InternVL2_5-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenGVLab/InternVL2_5-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/OpenGVLab/InternVL2_5-4B

SGLang

How to use OpenGVLab/InternVL2_5-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenGVLab/InternVL2_5-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenGVLab/InternVL2_5-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenGVLab/InternVL2_5-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenGVLab/InternVL2_5-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use OpenGVLab/InternVL2_5-4B with Docker Model Runner:
```
docker model run hf.co/OpenGVLab/InternVL2_5-4B
```

czczup commited on Dec 6, 2024

Commit

6b764ff

verified ·

1 Parent(s): 786e639

Delete eval_llm_benchmark.log

Browse files

Files changed (1) hide show

eval_llm_benchmark.log +0 -61

eval_llm_benchmark.log DELETED Viewed

@@ -1,61 +0,0 @@
-/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl_eval2/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
-  warn("The installed version of bitsandbytes was compiled without GPU support. "
-/mnt/petrelfs/wangweiyun/miniconda3/envs/internvl_eval2/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
-model path is /mnt/petrelfs/wangweiyun/workspace_cz/InternVL/internvl_chat_dev/share_internvl/InternVL2_5-4B
-11/21 15:12:04 - OpenCompass - WARNING - No previous results to reuse!
-11/21 15:12:04 - OpenCompass - INFO - Reusing experiements from 20241121_151204
-11/21 15:12:04 - OpenCompass - INFO - Current exp folder: /mnt/petrelfs/wangweiyun/workspace_cz/InternVL/internvl_chat_dev/share_internvl/InternVL2_5-4B/20241121_151204
-11/21 15:12:08 - OpenCompass - INFO - Partitioned into 256 tasks.
-[                                                  ] 0/256, elapsed: 0s, ETA:
-11/21 16:04:08 - OpenCompass - INFO - Partitioned into 287 tasks.
-[                                                  ] 0/287, elapsed: 0s, ETA:
-dataset                       version    metric                        mode    internvl-chat-20b
-----------------------------  ---------  ----------------------------  ------  -------------------
-mmlu                          -          naive_average                 gen     17.00
-mmlu_pro                      -          -                             -       -
-cmmlu                         -          naive_average                 gen     17.71
-ceval                         -          naive_average                 gen     17.43
-agieval                       -          -                             -       -
-GaokaoBench                   -          weighted_average              gen     18.45
-GPQA_extended                 -          -                             -       -
-GPQA_main                     -          -                             -       -
-GPQA_diamond                  -          -                             -       -
-ARC-c                         -          -                             -       -
-truthfulqa                    -          -                             -       -
-triviaqa                      2121ce     score                         gen     14.79
-triviaqa_wiki_1shot           -          -                             -       -
-nq                            3dcea1     score                         gen     5.60
-C3                            8c358f     accuracy                      gen     22.90
-race-high                     9a54b6     accuracy                      gen     21.47
-flores_100                    -          -                             -       -
-winogrande                    b36770     accuracy                      gen     23.52
-hellaswag                     e42710     accuracy                      gen     21.30
-bbh                           -          naive_average                 gen     18.67
-gsm8k                         1d7fe4     accuracy                      gen     16.00
-math                          393424     accuracy                      gen     9.30
-TheoremQA                     6f0af8     score                         gen     5.25
-MathBench                     -          -                             -       -
-openai_humaneval              8e312c     humaneval_pass@1              gen     17.68
-humaneval_plus                -          -                             -       -
-humanevalx                    -          -                             -       -
-sanitized_mbpp                a447ff     score                         gen     16.34
-mbpp_plus                     -          -                             -       -
-mbpp_cn                       6fb572     score                         gen     12.20
-leval                         -          -                             -       -
-leval_closed                  -          -                             -       -
-leval_open                    -          -                             -       -
-longbench                     -          -                             -       -
-longbench_single-document-qa  -          -                             -       -
-longbench_multi-document-qa   -          -                             -       -
-longbench_summarization       -          -                             -       -
-longbench_few-shot-learning   -          -                             -       -
-longbench_synthetic-tasks     -          -                             -       -
-longbench_code-completion     -          -                             -       -
-teval                         -          -                             -       -
-teval_zh                      -          -                             -       -
-IFEval                        3321a3     Prompt-level-strict-accuracy  gen     21.81
-IFEval                        3321a3     Inst-level-strict-accuracy    gen     34.17
-IFEval                        3321a3     Prompt-level-loose-accuracy   gen     23.29
-IFEval                        3321a3     Inst-level-loose-accuracy     gen     35.73
-11/21 16:08:33 - OpenCompass - INFO - write summary to /mnt/petrelfs/wangweiyun/workspace_cz/InternVL/internvl_chat_dev/share_internvl/InternVL2_5-4B/20241121_151204/summary/summary_20241121_151204.txt
-11/21 16:08:33 - OpenCompass - INFO - write csv to /mnt/petrelfs/wangweiyun/workspace_cz/InternVL/internvl_chat_dev/share_internvl/InternVL2_5-4B/20241121_151204/summary/summary_20241121_151204.csv