Instructions to use docling-project/SmolDocling-256M-preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use docling-project/SmolDocling-256M-preview with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="docling-project/SmolDocling-256M-preview") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForImageTextToText processor = AutoProcessor.from_pretrained("docling-project/SmolDocling-256M-preview") model = AutoModelForImageTextToText.from_pretrained("docling-project/SmolDocling-256M-preview") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use docling-project/SmolDocling-256M-preview with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "docling-project/SmolDocling-256M-preview" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "docling-project/SmolDocling-256M-preview", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/docling-project/SmolDocling-256M-preview
- SGLang
How to use docling-project/SmolDocling-256M-preview with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "docling-project/SmolDocling-256M-preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "docling-project/SmolDocling-256M-preview", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "docling-project/SmolDocling-256M-preview" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "docling-project/SmolDocling-256M-preview", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use docling-project/SmolDocling-256M-preview with Docker Model Runner:
docker model run hf.co/docling-project/SmolDocling-256M-preview
Upload ONNX weights
#7
by Xenova HF Staff - opened
- config.json +6 -0
- onnx/decoder_model_merged.onnx +3 -0
- onnx/decoder_model_merged_bnb4.onnx +3 -0
- onnx/decoder_model_merged_fp16.onnx +3 -0
- onnx/decoder_model_merged_int8.onnx +3 -0
- onnx/decoder_model_merged_q4.onnx +3 -0
- onnx/decoder_model_merged_q4f16.onnx +3 -0
- onnx/decoder_model_merged_quantized.onnx +3 -0
- onnx/decoder_model_merged_uint8.onnx +3 -0
- onnx/embed_tokens.onnx +3 -0
- onnx/embed_tokens_bnb4.onnx +3 -0
- onnx/embed_tokens_fp16.onnx +3 -0
- onnx/embed_tokens_int8.onnx +3 -0
- onnx/embed_tokens_q4.onnx +3 -0
- onnx/embed_tokens_q4f16.onnx +3 -0
- onnx/embed_tokens_quantized.onnx +3 -0
- onnx/embed_tokens_uint8.onnx +3 -0
- onnx/vision_encoder.onnx +3 -0
- onnx/vision_encoder_bnb4.onnx +3 -0
- onnx/vision_encoder_fp16.onnx +3 -0
- onnx/vision_encoder_int8.onnx +3 -0
- onnx/vision_encoder_q4.onnx +3 -0
- onnx/vision_encoder_q4f16.onnx +3 -0
- onnx/vision_encoder_quantized.onnx +3 -0
- onnx/vision_encoder_uint8.onnx +3 -0
config.json
CHANGED
|
@@ -116,6 +116,12 @@
|
|
| 116 |
"tie_word_embeddings": false,
|
| 117 |
"torch_dtype": "bfloat16",
|
| 118 |
"transformers_version": "4.50.0.dev0",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
"use_cache": true,
|
| 120 |
"vision_config": {
|
| 121 |
"hidden_size": 768,
|
|
|
|
| 116 |
"tie_word_embeddings": false,
|
| 117 |
"torch_dtype": "bfloat16",
|
| 118 |
"transformers_version": "4.50.0.dev0",
|
| 119 |
+
"transformers.js_config": {
|
| 120 |
+
"kv_cache_dtype": {
|
| 121 |
+
"q4f16": "float16",
|
| 122 |
+
"fp16": "float16"
|
| 123 |
+
}
|
| 124 |
+
},
|
| 125 |
"use_cache": true,
|
| 126 |
"vision_config": {
|
| 127 |
"hidden_size": 768,
|
onnx/decoder_model_merged.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c75155e5e121b9d89bf5435d396ccd9e817572d8a0e4589e1c0eb5574bb3c69e
|
| 3 |
+
size 540972236
|
onnx/decoder_model_merged_bnb4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:51ef8fa60e201a412c1a0b24333bfa2a6c73d4f45e58b9e2411866cfc6fc707b
|
| 3 |
+
size 78486831
|
onnx/decoder_model_merged_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ccd0761dc1e061c706381fb345a01977d4ec4db2ee97d3acb6886f65fbe2ba81
|
| 3 |
+
size 270764878
|
onnx/decoder_model_merged_int8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9fdce1ec841ad5b730d2c220ff2147f2114cecf7b5518c27cc050c29bf75e997
|
| 3 |
+
size 137553254
|
onnx/decoder_model_merged_q4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ef6045cf8e613277a745f5b56eaf1cc2dda00c64d984acaa005fa5ef706efcc5
|
| 3 |
+
size 86894835
|
onnx/decoder_model_merged_q4f16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e9cda6976be0f2b446a01ace90aa04a520116b4641e544cbc74dd8a70335b606
|
| 3 |
+
size 77385255
|
onnx/decoder_model_merged_quantized.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88d2ef3239d292224065443e7dc6a8b87195db94a38280e23bd4b5ca18c1d3b5
|
| 3 |
+
size 137553365
|
onnx/decoder_model_merged_uint8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88d2ef3239d292224065443e7dc6a8b87195db94a38280e23bd4b5ca18c1d3b5
|
| 3 |
+
size 137553365
|
onnx/embed_tokens.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:816939a02a48a540330655f6cc1a33cb47b2021aec0ebb356a016c50b826f3f6
|
| 3 |
+
size 113541419
|
onnx/embed_tokens_bnb4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7b6ad5d0ac29dc4bd893b61606f760fac40481a4eecbcb318502b05e3562aa37
|
| 3 |
+
size 113541438
|
onnx/embed_tokens_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f08a824993fe5ea9d5affd34a986a3700c4c5cb0fd21993182c03a58c4a0ee3a
|
| 3 |
+
size 56770887
|
onnx/embed_tokens_int8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:74e69c670cb770eb3b430c7fdd770a9e4e0f174edef568e6aa67df4276c398e5
|
| 3 |
+
size 28385824
|
onnx/embed_tokens_q4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7b6ad5d0ac29dc4bd893b61606f760fac40481a4eecbcb318502b05e3562aa37
|
| 3 |
+
size 113541438
|
onnx/embed_tokens_q4f16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:664289fb111a87112cf97c9ac4d9122119740bc5da3a6eb8305fa1063dec65a4
|
| 3 |
+
size 56770906
|
onnx/embed_tokens_quantized.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:74e69c670cb770eb3b430c7fdd770a9e4e0f174edef568e6aa67df4276c398e5
|
| 3 |
+
size 28385824
|
onnx/embed_tokens_uint8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:74e69c670cb770eb3b430c7fdd770a9e4e0f174edef568e6aa67df4276c398e5
|
| 3 |
+
size 28385824
|
onnx/vision_encoder.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9c97aafac0543632c778938e2870afb73a81c7cc3a3e01e84b43a096bd668820
|
| 3 |
+
size 374316454
|
onnx/vision_encoder_bnb4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fcc3b8bf6e0b12be05c5ce66adc5d504fb0efb4a00dbfc53a8f8dbc1cca1516e
|
| 3 |
+
size 58034696
|
onnx/vision_encoder_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f86f88e185e231b91232147206771e267d8ab038aa4e30fb29e376d9ec4db4fb
|
| 3 |
+
size 187295327
|
onnx/vision_encoder_int8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3251860101c63f90c44ec475c874346c461de48421aacc13b5f5741004282240
|
| 3 |
+
size 94247884
|
onnx/vision_encoder_q4.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fbcfa73fba2477b564a364123bc066922c9e692a9389c24eb904009f922e9aa0
|
| 3 |
+
size 63784944
|
onnx/vision_encoder_q4f16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c0e3c83edfaf3c946e592f50d9b238eb422c0c4ef62643883a46d7c5ebb7284a
|
| 3 |
+
size 55038218
|
onnx/vision_encoder_quantized.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b5e42cf6c8719dba60582c3722cba72673de5e2b739c01e5147bc34d9193ae20
|
| 3 |
+
size 94247926
|
onnx/vision_encoder_uint8.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b5e42cf6c8719dba60582c3722cba72673de5e2b739c01e5147bc34d9193ae20
|
| 3 |
+
size 94247926
|