Instructions to use microsoft/Phi-3-vision-128k-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-vision-128k-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-vision-128k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-vision-128k-instruct", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use microsoft/Phi-3-vision-128k-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-vision-128k-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-vision-128k-instruct

SGLang

How to use microsoft/Phi-3-vision-128k-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-vision-128k-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-vision-128k-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-vision-128k-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-vision-128k-instruct
```

the model can't stop generating for content extraction

by tankstarwar - opened May 22, 2024

Discussion

tankstarwar

May 22, 2024

I asked the model to extract tabular content from a 3x5 area of a spreadsheet image, but the model seems never stopped and went OOM eventually, is it an issue?

donniems

May 22, 2024

@tankstarwar We have noticed this looping issue and our language team is actively solving it.

Alex01837178373

May 24, 2024

I have the same thing, but to describe the photo with tags

nguyenbh

Microsoft org May 24, 2024

@tankstarwar @Alex01837178373 Can you share your image and prompt?

Alex01837178373

May 24, 2024

@tankstarwar @Alex01837178373 Can you share your image and prompt?

Alex01837178373

May 24, 2024

@tankstarwar @Alex01837178373 Can you share your image and prompt?

nguyenbh

Microsoft org May 24, 2024

Thanks for sharing! The Phi-3 Vision model is intended to use in English. In the cases above it looks like the text inputs are in Russian.

Alex01837178373

May 25, 2024

Thanks for sharing! The Phi-3 Vision model is intended to use in English. In the cases above it looks like the text inputs are in Russian.

In this case, I used Phi 3 vision to write captions to the photo in the form of tags, I used English language

tankstarwar

May 28, 2024

@tankstarwar @Alex01837178373 Can you share your image and prompt?

Hi Alex,

I used this the dummy tabular image created in spreadsheet.

And single prompt like:
{"role": "user", "content": "<|image_1|>\nextract the sales data from the table above and output in json format."}

nguyenbh

Microsoft org Jun 17, 2024

Thanks for sharing your example.
I am trying the example on Azure AI https://ai.azure.com/explore/models/Phi-3-vision-128k-instruct/version/2/registry/azureml

It looks pretty reasonable.

tankstarwar

Jun 20, 2024

Thanks for sharing your example.
I am trying the example on Azure AI https://ai.azure.com/explore/models/Phi-3-vision-128k-instruct/version/2/registry/azureml

It looks pretty reasonable.

OK thanks for the feedback, I tried AzureML version and it does work. Previously, I was testing it with some Huggingface space and also on my local machine (no CUDA so disabled flash attention) and got the issue, it could be something wrong with the environment setup I guess.

nguyenbh changed discussion status to closed Jul 9, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment