Instructions to use HuggingFaceTB/SmolVLM-500M-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceTB/SmolVLM-500M-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HuggingFaceTB/SmolVLM-500M-Instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")
model = AutoModelForImageTextToText.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HuggingFaceTB/SmolVLM-500M-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceTB/SmolVLM-500M-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceTB/SmolVLM-500M-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/HuggingFaceTB/SmolVLM-500M-Instruct

SGLang

How to use HuggingFaceTB/SmolVLM-500M-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceTB/SmolVLM-500M-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceTB/SmolVLM-500M-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceTB/SmolVLM-500M-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceTB/SmolVLM-500M-Instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use HuggingFaceTB/SmolVLM-500M-Instruct with Docker Model Runner:
```
docker model run hf.co/HuggingFaceTB/SmolVLM-500M-Instruct
```

Upload ONNX weights

by Xenova HF Staff - opened Jan 22, 2025

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+72

-0

Files changed (24) hide show

onnx/decoder_model_merged.onnx +3 -0
onnx/decoder_model_merged_bnb4.onnx +3 -0
onnx/decoder_model_merged_fp16.onnx +3 -0
onnx/decoder_model_merged_int8.onnx +3 -0
onnx/decoder_model_merged_q4.onnx +3 -0
onnx/decoder_model_merged_q4f16.onnx +3 -0
onnx/decoder_model_merged_quantized.onnx +3 -0
onnx/decoder_model_merged_uint8.onnx +3 -0
onnx/embed_tokens.onnx +3 -0
onnx/embed_tokens_bnb4.onnx +3 -0
onnx/embed_tokens_fp16.onnx +3 -0
onnx/embed_tokens_int8.onnx +3 -0
onnx/embed_tokens_q4.onnx +3 -0
onnx/embed_tokens_q4f16.onnx +3 -0
onnx/embed_tokens_quantized.onnx +3 -0
onnx/embed_tokens_uint8.onnx +3 -0
onnx/vision_encoder.onnx +3 -0
onnx/vision_encoder_bnb4.onnx +3 -0
onnx/vision_encoder_fp16.onnx +3 -0
onnx/vision_encoder_int8.onnx +3 -0
onnx/vision_encoder_q4.onnx +3 -0
onnx/vision_encoder_q4f16.onnx +3 -0
onnx/vision_encoder_quantized.onnx +3 -0
onnx/vision_encoder_uint8.onnx +3 -0

onnx/decoder_model_merged.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d4dc5a8464c92cf151be233e23da809992e111f5596ba245fe1eca28e838b7b
+size 1450425745

onnx/decoder_model_merged_bnb4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c3b311d0d34a7c067485ad9d440045c892a73885c7db52f13336b428d5c302e3
+size 206503276

onnx/decoder_model_merged_fp16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1dc67f9d505c4f0a1a9de1626537aa2fd440fbb16b6a74abfddf66169e1e486e
+size 725489060

onnx/decoder_model_merged_int8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:36f53411b56c21868b0cf3d1fa0c1a7395ab88bcd1f7ae677bf91c54ab09cca0
+size 365038968

onnx/decoder_model_merged_q4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d64c0fdce1c3515c40a08ff8e398bfbc0b2e961ae46d1b84b91e6a7b269b9d1
+size 229119140

onnx/decoder_model_merged_q4f16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:289540c10892e905107f2a6f07a21d71df1aa3d73e0d3432d793df2ad80850e3
+size 205328183

onnx/decoder_model_merged_quantized.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69f48213b59637eee88e2b1af3c6bb23677fce77774a7237e8c1516c6b78ce47
+size 365039089

onnx/decoder_model_merged_uint8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69f48213b59637eee88e2b1af3c6bb23677fce77774a7237e8c1516c6b78ce47
+size 365039089

onnx/embed_tokens.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6fe00a73de387cbe52cdb78f1b7c8bcda6bcb9974e40dc1b143315fc9f7e79af
+size 189235499

onnx/embed_tokens_bnb4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:19f4fad7cb17c6dad11b4520d2abcba173fc91f755d71907d31d11a9e5761606
+size 189235518

onnx/embed_tokens_fp16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6b5f8cb6f78f90182bb60a3df22e9c47c93dc771d8d8172cce1f653f8da8a746
+size 94617986

onnx/embed_tokens_int8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344

onnx/embed_tokens_q4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:19f4fad7cb17c6dad11b4520d2abcba173fc91f755d71907d31d11a9e5761606
+size 189235518

onnx/embed_tokens_q4f16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76c3b8232c463118190d1c711f2a7bff00a72f2077b9b19998f3a2c6344eb035
+size 94618005

onnx/embed_tokens_quantized.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344

onnx/embed_tokens_uint8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344

onnx/vision_encoder.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7ecd87031065252c5e3419f9a75db938a92cd5d55d2c22024d8815efd980104f
+size 393190865

onnx/vision_encoder_bnb4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e4af9028ddd3357b50a4d823987bb81bf2aca83eeba425bfc02aaa339eeca9f4
+size 60688947

onnx/vision_encoder_fp16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b2fca5aecb5fe29549ce0a1cc3091f64f7f33dafb5f5f215e2b774b5cf1eeec3
+size 196731877

onnx/vision_encoder_int8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:50b814aa048a996bf5d5128a16c19046371a9a1aae24b930672f550af6c43646
+size 98966519

onnx/vision_encoder_q4.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:179812d5b1315edbe3e7164ea3d847c52723692e83ac9c770a5234962663fcad
+size 66734107

onnx/vision_encoder_q4f16.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d5f3df9fd7113e564b1e167de20750572d25fd75c822365f496ff467bc9e7457
+size 57691792

onnx/vision_encoder_quantized.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0612873758c96d759910a45dddbd150ea26d40cffb1ae49f391ea39dcbf50c09
+size 98966563

onnx/vision_encoder_uint8.onnx ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0612873758c96d759910a45dddbd150ea26d40cffb1ae49f391ea39dcbf50c09
+size 98966563