Instructions to use microsoft/Phi-3-small-8k-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-small-8k-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-small-8k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-small-8k-instruct", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Phi-3-small-8k-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-small-8k-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-small-8k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-small-8k-instruct

SGLang

How to use microsoft/Phi-3-small-8k-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-small-8k-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-small-8k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-small-8k-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-small-8k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-small-8k-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-small-8k-instruct
```

Why a different architecture from mini and medium?

by winddude - opened May 23, 2024

Discussion

winddude

May 23, 2024

How come the small family of models have a different architecture from the mini and medium? Phi3SmallForCausalLM vs Phi3ForCausalLM.

Trangle

May 23, 2024

The same question. Can you explain the specific considerations and advantages? Or is it simply an early experiment

alexcardo

May 23, 2024

Because of this, there is no possibility to quantize this model. Please load the GGUF quants or explain it to us how to quantize it. Not every of us have an A100 at home. As for me, I'm very excited with Phi-3-Mini, and I suspect that Phi-3-small-q4_K_M could ideally fit my Macbook M1 8G.

Please, help us! I've been talking with a lot of people, nobody managed to quantize this model!

bapatra

Microsoft org May 23, 2024

Hi !
There are a few reasons for the design choices.

The tiktoken based tokenizer and larger vocab gave us some performance gains in our preliminary experiments. Additionally, tiktoken had some performance gains compared to the transformers FastTokenizers (see the tiktoken repo for a benchmark).
We tried to gear the 7B model towards faster inference. As a result, the model uses block-sparse attention in conjunction with dense attention in addition to GQA. This reduces the kv-cache memory footprint considerably thereby allowing for faster inference on a continuous batcher like vLLM. We have an open PR with vLLM for integrating the blocksparse kernels there as well (this PR).

Unfortunately, because of the custom kernels, we've not been able to leverage the open-source GGUF formats from llama.cpp (as well as the subsequent quantizations it offers). However, there is active work going on to get the model onto llama.cpp (see this issue). Once that is done, the quantized models should follow :)

yutsai123

May 23, 2024

@bapatra , is this the reason that I saw the inferencing issue in this link? Thanks!

bapatra

Microsoft org May 24, 2024

I don't think the two issues are related. Commented on the question there !

bapatra changed discussion status to closed May 24, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment