Instructions to use moonshotai/Kimi-K2.6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moonshotai/Kimi-K2.6 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="moonshotai/Kimi-K2.6", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("moonshotai/Kimi-K2.6", trust_remote_code=True, dtype="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use moonshotai/Kimi-K2.6 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moonshotai/Kimi-K2.6"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K2.6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/moonshotai/Kimi-K2.6

SGLang

How to use moonshotai/Kimi-K2.6 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moonshotai/Kimi-K2.6" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K2.6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moonshotai/Kimi-K2.6" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K2.6",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use moonshotai/Kimi-K2.6 with Docker Model Runner:
```
docker model run hf.co/moonshotai/Kimi-K2.6
```

fix: add missing use_deterministic_attn parameter to MoonViT3dEncoder

#22

by ace-coreweave - opened Apr 24

base: refs/heads/main

←

from: refs/pr/22

Discussion Files changed

+19

-4

ace-coreweave

Apr 24

MoonViT3dEncoder.init references self.use_deterministic_attn on line 575
when constructing the MoonViTEncoderLayer blocks, but the attribute is never
set on self. Loading the model via AutoModelForCausalLM with
trust_remote_code=True raises:

AttributeError: 'MoonViT3dEncoder' object has no attribute
                'use_deterministic_attn'

The sibling class MoonViTEncoderLayer already accepts use_deterministic_attn
as a keyword parameter with default False, so the attribute on the parent
3d-encoder was clearly intended to plumb through the same flag. Restore the
missing parameter with the same default.

Production serving paths (vLLM's Kimi-K25 model executor) bypass the HF
custom modeling init and construct the vision tower differently, so this
bug is invisible at serving time but blocks transformers-based workflows
like ModelOpt NVFP4 quantization and HF-native fine-tuning.

Identical fix already merged in Kimi-K2.5 PR #91 (by @katuni4ka , approved
by @fxmarty-amd ). This mirrors it to K2.6 byte-for-byte.

Minimal repro:

from transformers import AutoModelForCausalLM
AutoModelForCausalLM.from_pretrained(
    "moonshotai/Kimi-K2.6", trust_remote_code=True, torch_dtype="auto",
)

fix: add missing use_deterministic_attn parameter to MoonViT3dEncoderf13e4d07

bdellabe

28 days ago

This resolved my issue, @bigeagle can we merge this in so we don't require other users to manually update?

bigeagle

Moonshot AI org 27 days ago

thanks for your contribution!

also fix weight init issuebed55d49

bigmoyan

Moonshot AI org 27 days ago

@bdellabe @ace-coreweave Hi, I've also added some code to fix the weight initialization issue. AutoModelForCausalLM.from_pretrained now works on my end. However, this doesn't mean transformers inference is fully supported — if you plan to implement Kimi k2.6 inference in other frameworks, please mainly refer to the vLLM/SGLang implementation.

bigmoyan changed pull request status to merged 27 days ago

bigmoyan

Moonshot AI org 26 days ago

@ace-coreweave @bdellabe see: https://huggingface.co/moonshotai/Kimi-K2.6/discussions/26

fxmarty-amd

19 days ago

@bigmoyan Any plan to have compatibility with Transformers v5?

  File "/root/.cache/huggingface/modules/transformers_modules/moonshotai/Kimi_hyphen_K2_dot_6/2755962d07cb42aa2d988a35bcb65cd4a9c2de82/modeling_deepseek.py", line 47, in <module>
    from transformers.utils.import_utils import is_torch_fx_available
ImportError: cannot import name 'is_torch_fx_available' from 'transformers.utils.import_utils' (/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py). Did you mean: 'is_torch_available'?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment