Instructions to use vllm-project/Kimi-K3.1-Pro-pruned75 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vllm-project/Kimi-K3.1-Pro-pruned75 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="vllm-project/Kimi-K3.1-Pro-pruned75", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("vllm-project/Kimi-K3.1-Pro-pruned75", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vllm-project/Kimi-K3.1-Pro-pruned75 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vllm-project/Kimi-K3.1-Pro-pruned75"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vllm-project/Kimi-K3.1-Pro-pruned75",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/vllm-project/Kimi-K3.1-Pro-pruned75

SGLang

How to use vllm-project/Kimi-K3.1-Pro-pruned75 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vllm-project/Kimi-K3.1-Pro-pruned75" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vllm-project/Kimi-K3.1-Pro-pruned75",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vllm-project/Kimi-K3.1-Pro-pruned75" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vllm-project/Kimi-K3.1-Pro-pruned75",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use vllm-project/Kimi-K3.1-Pro-pruned75 with Docker Model Runner:
```
docker model run hf.co/vllm-project/Kimi-K3.1-Pro-pruned75
```

Kimi-K3.1-Pro-pruned75

A calibration-free, expert-pruned checkpoint derived from moonshotai/Kimi-K3.1-Pro that removes 75% of the routed experts in every MoE layer. It is intended for vLLM serving and evaluation smoke tests on fewer GPUs, not for quality-sensitive use. Expect a large quality regression relative to the source model.

Pruning

Routed experts per MoE layer: 896 → 224 (75% removed), across 92 MoE layers
Ranking: signed sum of each expert's router row, sum(router.weight[i]), computed independently per layer
Removed: pruned expert tensors, plus the matching rows of gate.weight and entries of gate.e_score_correction_bias. Expert indices are renumbered and text_config.num_experts is set to 224
Unchanged: attention, shared experts, dense layer, vision tower, and the original MXFP4 quantization of the routed experts
Calibration data: none
Size: ~443 GB (vs. ~1.5 TB for the source)

Produced with the MagnitudeExpertPruner converter in compressed-tensors, with its default L1 scoring (weight.abs().sum(-1)) replaced by a signed sum. On this model the signed ranking scored much higher (GSM8K 43.7% vs. 14.4% for L1).

import torch
from compressed_tensors.entrypoints.convert import (
    MagnitudeExpertPruner,
    convert_checkpoint,
)
from compressed_tensors.entrypoints.convert.converters import (
    magnitude_expert_pruner as mep,
)
from safetensors import safe_open


def signed_sum_retained(router_names, weight_map, model_files, sparsity):
    retained = {}
    for name in router_names:
        with safe_open(model_files[weight_map[name]], framework="pt") as f:
            scores = f.get_tensor(name).float().sum(dim=-1)
        k = max(1, scores.shape[0] - round(sparsity * scores.shape[0]))
        retained[name] = sorted(torch.topk(scores, k).indices.tolist())
    return retained


mep._compute_retained_experts = signed_sum_retained

MODEL_ID = "moonshotai/Kimi-K3.1-Pro"
convert_checkpoint(
    model_stub=MODEL_ID,
    save_directory="Kimi-K3.1-Pro-pruned75",
    converter=MagnitudeExpertPruner.from_pretrained(
        MODEL_ID,
        router_pattern=r"language_model\..*\.block_sparse_moe\.gate\.weight$",
        expert_pattern=r"language_model\..*\.block_sparse_moe\.experts\.\d+\.w[123]\.",
        aux_pattern=r"block_sparse_moe\.gate\.e_score_correction_bias$",
        sparsity=0.75,
        num_experts_config_key="num_experts",
    ),
    device="cpu",
)

Evaluation

GSM8K 5-shot via lm_eval (local-completions) against vLLM, TP=4 on NVIDIA B300:

Metric	Value
exact_match (flexible-extract)	43.67% ± 1.37
exact_match (strict-match)	43.06% ± 1.36

Usage

vllm serve vllm-project/Kimi-K3.1-Pro-pruned75 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

License

Same as the source model: see the Kimi K3.1 Pro license.

Downloads last month: 115

Safetensors

Model size

738B params

Tensor type

BF16

F32

Model tree for vllm-project/Kimi-K3.1-Pro-pruned75

Base model

moonshotai/Kimi-K3.1-Pro

Finetuned

(11)

this model

Spaces using vllm-project/Kimi-K3.1-Pro-pruned75 4

Evaluation results

Accuracy on GSM8K
test set self-reported

43.670