Instructions to use microsoft/Phi-3-medium-128k-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-medium-128k-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-medium-128k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-medium-128k-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-medium-128k-instruct", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use microsoft/Phi-3-medium-128k-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-medium-128k-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-medium-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-medium-128k-instruct

SGLang

How to use microsoft/Phi-3-medium-128k-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-medium-128k-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-medium-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-medium-128k-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-medium-128k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-medium-128k-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-medium-128k-instruct
```

Clarification requested: using locally generated Phi-3 outputs as training data

#21

by RyoSpiralArchitect - opened Aug 30

Discussion

RyoSpiralArchitect

Aug 30

Hugging Face CLI

Hello Microsoft Phi team,

I am evaluating microsoft/Phi-3-medium-128k-instruct at revision a088b37c71d441ab6d862bb3fcfe6165b3014702 and would appreciate a model-specific licensing clarification.

The proposed workflow is:

download the MIT-licensed model weights from this repository and run inference entirely on local hardware;
do not use Azure, Microsoft Foundry, or another Microsoft-hosted AI service;
use prompts and source material for which we are responsible;
retain some locally generated text as synthetic examples in the training corpus of a separately developed model called Aether; and
do not attempt to recover Phi weights, hidden system instructions, or other proprietary internals.

Could a Microsoft model maintainer please clarify:

Does Microsoft regard this local use of Phi-3-generated text as permitted under the license and terms distributed with this open model?
Do the output-training restrictions in Microsoft's AI-service terms apply when the downloaded open weights are run fully locally without using a Microsoft service?
If the repository's MIT license does not affirmatively cover generated output, is there another Microsoft term or official source that governs this use, or would separate permission be required?

A yes/no/conditional answer with a link to the applicable Microsoft terms would be especially helpful. I understand that Microsoft cannot determine third-party rights in particular outputs or provide advice about every jurisdiction; this question is limited to Microsoft's intended permission and contractual scope for the locally run open model.

Thank you.

deleted

Sep 4

•

This comment has been hidden

deleted

Sep 5

•

This comment has been hidden

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment