Instructions to use HuggingFaceM4/Idefics3-8B-Llama3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/Idefics3-8B-Llama3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HuggingFaceM4/Idefics3-8B-Llama3")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3")
model = AutoModelForMultimodalLM.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HuggingFaceM4/Idefics3-8B-Llama3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/Idefics3-8B-Llama3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/Idefics3-8B-Llama3

SGLang

How to use HuggingFaceM4/Idefics3-8B-Llama3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/Idefics3-8B-Llama3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/Idefics3-8B-Llama3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use HuggingFaceM4/Idefics3-8B-Llama3 with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/Idefics3-8B-Llama3
```

Image encoding / rescaling Question

#11

by ayyylemao - opened Aug 21, 2024

Discussion

ayyylemao

Aug 21, 2024

•

edited Aug 21, 2024

You can choose the default resolution the images will be rescaled to by adding size= {"longest_edge": N*364} when initializing the processor (AutoProcessor.from_pretrained), with N your desired value. N=4 works best in practice (this is the default value), but for very large images, it could be interesting to pass N=5. This will have an impact on the number of visual tokens passed to the language model. If you are GPU-memory-constrained, you can decrease N, and choose for example N=3 or N=2, especially for low resolution images.

How is this to be understood ?

In the case of a square image, does it mean that if longest_edge setting is N=1, that the image gets down scaled to 364xX and the embedding only uses 169 visual tokens for the whole image?
If we use N=5, does this result in the original image being upscaled/downscaled to 1820x1820px (again square image source) and divided into 25 364x364 subimages or does no upscaling happen if a smaller image gets submitted than longest_edge specifies?
What about uneven aspect ratios? Are the images squashed into a 1:1 aspect ratio?

PS: I tried going above N=5 but there seems to be a hard coded limit. It might be nice to test N=6 for OCR tasks.

HugoLaurencon

Aug 23, 2024

•

edited Aug 23, 2024

In any case, it’s not going to respect the original aspect ratio. However, it might not be too far from it if N is not too small.

By default, for each image, we rescale the longest side L to N*364. To compute the other side of the image, we first compute it as if we wanted to respect the original ratio, and knowing the side L. Then, we round this number to the following multiple of 364. We can the resize an image.

Therefore, for N=5, in general you don’t have 1820x1820 images. Only one side will be 1820 and for the other it depends.

N=6 can work but might require further fine-tuning. We noticed a clear boost only on DocVQA with N=5 already compared to N=4

If there is a bug saying that you can’t go higher, it’s probably because the default is the max size at 5*364, but you can probably modify that by changing a parameter

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment