Instructions to use microsoft/Phi-3.5-vision-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3.5-vision-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="microsoft/Phi-3.5-vision-instruct", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3.5-vision-instruct", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Phi-3.5-vision-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3.5-vision-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-vision-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3.5-vision-instruct

SGLang

How to use microsoft/Phi-3.5-vision-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3.5-vision-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-vision-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3.5-vision-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3.5-vision-instruct",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3.5-vision-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3.5-vision-instruct
```

rageSpin commited on Oct 10, 2024

Commit

6da17bd

verified ·

1 Parent(s): 4a0d683

Update processing_phi3_v.py

Browse files

This model was not able to process list of numpy arrays (and probably with pytorch tensors)
you can try yourself in the example code on this model card. Just use 'np.array(Image.open(requests.get(url, stream=True).raw)) '

![image.png](https://cdn-uploads.huggingface.co/production/uploads/6421a0e5918f0fd889f0283c/nE1ZvC_mi21l2v72dzlh3.png)

I solved this issue implementing a function to convert any type of valid image to PIL Image object.

Files changed (1) hide show

processing_phi3_v.py +22 -4

processing_phi3_v.py CHANGED Viewed

@@ -56,7 +56,7 @@ logger = logging.get_logger(__name__)
 if is_vision_available():
     from PIL import Image
-import torch
 import torchvision
 def padding_336(b):
@@ -205,6 +205,22 @@ class Phi3VImageProcessor(BaseImageProcessor):
         num_img_tokens = int((new_height // 336 * new_width // 336 + 1) * 144 + 1 + (new_height // 336 + 1) * 12)
         return num_img_tokens
     def preprocess(
         self,
         images: ImageInput,
@@ -245,8 +261,8 @@ class Phi3VImageProcessor(BaseImageProcessor):
                 "torch.Tensor, tf.Tensor or jax.ndarray."
             )
-        if do_convert_rgb:
-            images = [convert_to_rgb(image) for image in images]
         image_sizes = []
         img_processor = torchvision.transforms.Compose([
@@ -256,7 +272,9 @@ class Phi3VImageProcessor(BaseImageProcessor):
         # PIL images
         # HD_transform pad images to size of multiiply of 336, 336
-        # convert to RGB first
         images = [image.convert('RGB') for image in images]
         elems = [HD_transform(im, hd_num = self.num_crops) for im in images]
         # tensor transform and normalize

 if is_vision_available():
     from PIL import Image
+import torch  # why is this imported twice?
 import torchvision
 def padding_336(b):
         num_img_tokens = int((new_height // 336 * new_width // 336 + 1) * 144 + 1 + (new_height // 336 + 1) * 12)
         return num_img_tokens
+    def convert_PIL(self, image):
+        """
+        Convert an image to a PIL Image object if it is not already one.
+        Args:
+            image (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`): The image to be converted. Can be a numpy array or a torch tensor or PIL object.
+        Returns:
+            A PIL Image object.
+        """
+            if not isinstance(image, Image.Image):
+                return torchvision.transforms.functional.to_pil_image(image)
+            else:
+                return image
     def preprocess(
         self,
         images: ImageInput,
                 "torch.Tensor, tf.Tensor or jax.ndarray."
             )
+        # if do_convert_rgb:
+            # images = [convert_to_rgb(image) for image in images]
         image_sizes = []
         img_processor = torchvision.transforms.Compose([
         # PIL images
         # HD_transform pad images to size of multiiply of 336, 336
+        # check and convert if the images are in PIL format
+        images = [convert_PIL(image) for image in images]
+        # convert to RGB first (I think the argument "do_convert_rgb is useless, since it is forced here")
         images = [image.convert('RGB') for image in images]
         elems = [HD_transform(im, hd_num = self.num_crops) for im in images]
         # tensor transform and normalize