Instructions to use microsoft/Phi-3-vision-128k-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-vision-128k-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-vision-128k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-vision-128k-instruct", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Phi-3-vision-128k-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-vision-128k-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-vision-128k-instruct

SGLang

How to use microsoft/Phi-3-vision-128k-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-vision-128k-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-vision-128k-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-vision-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-vision-128k-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-vision-128k-instruct
```

Unable to load the model

#10

by armandal - opened May 22, 2024

Discussion

armandal

May 22, 2024

Hi,
I'm following the tuto, installed all libs requiered (including flash_attn), but I cannot load the model:

Is there any mistake in the versions of libraries that are requiered by the tuto ?

thanks!

vfragoso

May 22, 2024

Maybe, try to install the latest flash attention: pip install flash-attn --no-build-isolation.

yjeong

May 22, 2024

I am facing the same problem even though I have already installed flash-attn.
The vision model is the only one not working for me.
Is anyone else experiencing this issue?

benjamintx

May 23, 2024

•

edited May 23, 2024

Running on Windows or without flash attention

To enable the model on these enviroment here are steps that you may consider to follow:

Step 1: comment flash attention import code in modeling_phi3_v.py from line 52 to line 56.

# if is_flash_attn_2_available():
#     from flash_attn import flash_attn_func, flash_attn_varlen_func
#     from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input  # noqa

#     _flash_supports_window_size = "window_size" in list(inspect.signature(flash_attn_func).parameters)

Step 2: change _"attn_implementation" from "flash_attention_2" to "eager" in config.json or disable flash attention when you create the model as below.

model = AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-vision-128k-instruct', device_map="cuda", trust_remote_code=True, torch_dtype="auto", _attn_implementation="eager")

armandal

May 23, 2024

Maybe, try to install the latest flash attention: pip install flash-attn --no-build-isolation.

thanks for this recommendation, but it did not work :(

yjeong

May 23, 2024

To enable the model on these enviroment here are steps that you may consider to follow:

Step 1: comment flash attention import code in modeling_phi3_v.py from line 52 to line 56.

Step 2: change _"attn_implementation" from "flash_attention_2" to "eager" in config.json or disable flash attention when you create the model as below.

model = AutoModelForCausalLM.from_pretrained('microsoft/Phi-3-vision-128k-instruct', device_map="cuda", trust_remote_code=True, torch_dtype="auto", _attn_implementation="eager")

This works for me to load the model! Thx for your help

Kukedlc

May 24, 2024

•

edited May 24, 2024

Here's a version that works with the changes mentioned above, Kukedlc/Phi-3-Vision-Win-snap.
https://huggingface.co/Kukedlc/Phi-3-Vision-Win-snap

You just need to run the Python code provided in the model card; the changes in the .py file and config.json are already made.

armandal

May 24, 2024

•

edited May 24, 2024

Here's a version that works with the changes mentioned above, Kukedlc/Phi-3-Vision-Win-snap.
https://huggingface.co/Kukedlc/Phi-3-Vision-Win-snap

You just need to run the Python code provided in the model card; the changes in the .py file and config.json are already made.

thanks! it works after having installed the following packages:

pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers.git@v4.40.2
pip install flash_attn==2.5.8
pip install numpy==1.24.4
pip install Pillow==10.3.0
pip install Requests==2.31.0
pip install torch==2.3.0
pip install torchvision==0.18.0
pip install accelerate

nguyenbh changed discussion status to closed May 31, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment