Instructions to use moonshotai/Kimi-K2-Instruct-0905 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use moonshotai/Kimi-K2-Instruct-0905 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="moonshotai/Kimi-K2-Instruct-0905", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("moonshotai/Kimi-K2-Instruct-0905", trust_remote_code=True, dtype="auto") - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use moonshotai/Kimi-K2-Instruct-0905 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "moonshotai/Kimi-K2-Instruct-0905" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "moonshotai/Kimi-K2-Instruct-0905", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/moonshotai/Kimi-K2-Instruct-0905
- SGLang
How to use moonshotai/Kimi-K2-Instruct-0905 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "moonshotai/Kimi-K2-Instruct-0905" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "moonshotai/Kimi-K2-Instruct-0905", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "moonshotai/Kimi-K2-Instruct-0905" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "moonshotai/Kimi-K2-Instruct-0905", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use moonshotai/Kimi-K2-Instruct-0905 with Docker Model Runner:
docker model run hf.co/moonshotai/Kimi-K2-Instruct-0905
Token count differences between hf transformers and Moonshot AI API
First of all, thank you for the amazing model.
I am seeing discrepancies in the number of prompt tokens when I tokenize using Transformers and when I send messages to Moonshot AI's hosted endpoint. I am worried this will change the performance of the model due to the prompts being different.
Here are my messages
[
{
"role": "user",
"content": "What is the current temperature of New York, San Francisco and Chicago?"
},
{
"role": "assistant",
"content": "I'll get the current temperature for all three cities for you.",
"tool_calls": [
{
"id": "functions.get_current_weather:0",
"index": 0,
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\": \"New York\", \"unit\": \"celsius\"}"
}
},
{
"id": "functions.get_current_weather:1",
"index": 1,
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\": \"San Francisco\", \"unit\": \"celsius\"}"
}
},
{
"id": "functions.get_current_weather:2",
"index": 2,
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\": \"Chicago\", \"unit\": \"celsius\"}"
}
}
]
},
{
"role": "tool",
"tool_call_id": "functions.get_current_weather:0",
"name": "get_current_weather",
"content": "{\"location\": \"New York\", \"temperature\": 18, \"unit\": \"celsius\", \"description\": \"partly cloudy\", \"humidity\": 62}"
},
{
"role": "tool",
"tool_call_id": "functions.get_current_weather:1",
"name": "get_current_weather",
"content": "{\"location\": \"San Francisco\", \"temperature\": 16, \"unit\": \"celsius\", \"description\": \"foggy\", \"humidity\": 78}"
},
{
"role": "tool",
"tool_call_id": "functions.get_current_weather:2",
"name": "get_current_weather",
"content": "{\"location\": \"Chicago\", \"temperature\": 12, \"unit\": \"celsius\", \"description\": \"overcast\", \"humidity\": 55}"
}
]
Here are my tools
[
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": [
"celsius",
"fahrenheit"
]
}
}
}
}
}
]
I tokenize using Transformers like so:
tokenizer = AutoTokenizer.from_pretrained("moonshotai/Kimi-K2-Instruct-0905", trust_remote_code=True)
ids = tokenizer.apply_chat_template(
messages, tools=tools, tokenize=True
)
print(len(ids))
I am calling the Moonshot API through OpenRouter like this:
curl -X POST "https://openrouter.ai/api/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-d "$(jq -n \
--argjson messages "$(cat k2-messages-with-tool-response.json)" \
--argjson tools "$(cat tools.json)" \
'{
"model": "moonshotai/kimi-k2-0905",
"messages": $messages,
"tools": $tools,
"provider": {"only":["moonshotai"]},
}')" | jq
Can you help me understand why Transformers shows this as 391 tokens and Moonshot AI API shows it as 335?
Thank you very much
In moonshot API, the tool descriptions are encoded in a more token-efficient typescript-like format. This conversion is too complex to put in chat_template.jinja so we use raw json-schema in transformers.
Thank you for the informative response!
Is the conversion function in your github repository?
nope, but it's a great idea, we'll discuss on this.