moonshotai

/

Kimi-K3

like 10.8k

Follow

Moonshot AI 17.2k

Following are the 2 scripts I use (generated):

run_node_sglang.sh

#!/bin/bash
# 在容器内由 srun 每节点执行一次。依赖外部传入:
#   NODE0_IP, MODEL_PATH, HOST_PORT, TP_SIZE, EP_SIZE, NNODES, MEM_FRACTION
set -euo pipefail

NODE_RANK=$SLURM_PROCID

# 跨节点 NIC 必须在所有 rank 上显式指定 (SGLang H200 multiNodeHints 要求),
# 各节点网卡名不同, 运行时探测: 取默认路由所在接口。容器镜像可能无 ip 命令,
# 故从 /sys 读取而非依赖 iproute2。
detect_iface() {
    local best="" max=0 sz
    for d in /sys/class/net/*; do
        n=$(basename "$d")
        case "$n" in lo|docker*|veth*|br-*|virbr*) continue ;; esac
        [ -e "$d/device" ] || continue
        read -r sz < "$d/speed" 2>/dev/null || sz=0
        [ "$sz" -gt "$max" ] 2>/dev/null && { max=$sz; best=$n; }
        [ -z "$best" ] && best=$n
    done
    echo "$best"
}
IFACE=$(detect_iface)
LOCAL_IP=$(cat /sys/class/net/"$IFACE"/address >/dev/null 2>&1 && hostname -I | awk '{print $1}')

export NCCL_SOCKET_IFNAME="$IFACE"
export GLOO_SOCKET_IFNAME="$IFACE"
export SGLANG_HOST_IP="$LOCAL_IP"

# EFA: 用宿主机挂进来的 libfabric + efa
# export LD_LIBRARY_PATH to include efa related libs if needed
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1

# Triton JIT 缓存放节点本地盘: 默认落到 $HOME 共享盘时, 多 rank 并发
# JIT 同一 kernel 的 tmp+rename 互踩, FileNotFoundError (job 36)
export TRITON_CACHE_DIR="/tmp/triton-cache-rank${NODE_RANK}"

# SGLang H200 unified/balanced 档指定的 env
export NCCL_MNNVL_ENABLE=1
export NCCL_CUMEM_ENABLE=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0

echo "[rank ${NODE_RANK}] host=$(hostname) iface=${IFACE} ip=${LOCAL_IP}"

exec sglang serve \
    --trust-remote-code \
    --model-path "$MODEL_PATH" \
    --tp-size "$TP_SIZE" \
    --nnodes "$NNODES" \
    --node-rank "$NODE_RANK" \
    --dist-init-addr "${NODE0_IP}:20000" \
    --ep-size "$EP_SIZE" \
    --moe-runner-backend marlin \
    --decode-attention-backend flashmla \
    --enable-symm-mem \
    --mem-fraction-static "$MEM_FRACTION" \
    --reasoning-parser kimi_k3 \
    --tool-call-parser kimi_k3 \
    --host 0.0.0.0 \
    --port "$HOST_PORT"

serve_2node_sglang.slurm

#!/bin/bash
#SBATCH --job-name=k3-sgl
#SBATCH --partition=p5e
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=96
#SBATCH --mem=0
#SBATCH --exclusive
#SBATCH --time=7-00:00:00
#SBATCH --output=/home/ubuntu/workspace/kimi-k3-deploy/logs/sgl_%j.out
#SBATCH --error=/home/ubuntu/workspace/kimi-k3-deploy/logs/sgl_%j.err

###############################################################################
# Kimi-K3 —— SGLang 引擎
#
# 参数全部取自 SGLang 官方 cookbook 的 h200 / unified / balanced 档
# (docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3.md):
#   TP16 + EP16, moe-runner-backend marlin, decode-attention-backend flashmla,
#   --enable-symm-mem, --mem-fraction-static 0.85
# H200 官方节点数就是 2×8, 与 vLLM 侧 job 31 同规模, 便于对比。
#
# 注意 (相对 vLLM 侧的已知情况):
#   - SGLang H200 档同样用 marlin, 所以每卡权重大概率仍是 ~130GB
#     (H200 无原生 MXFP4 硬件, 必须重打包成 W4A16, 体积涨 ~33%)
#   - 官方 mem-fraction-static 只有 0.85, 比我vLLM 侧的 0.97 保守很多;
#     若 KV cache 不够, 先往上调这个值, 再考虑降 context
#   - 三档 H200 配置在官方文档里均为 verified:false / in-progress,
#     成熟度与 vLLM recipe 的 H200 相当, 不是"更稳"的选择
#
###############################################################################

set -euo pipefail

export MODEL_PATH="/home/ubuntu/models/Kimi-K3"
export HOST_PORT=30000
export TP_SIZE=16
export EP_SIZE=16
export NNODES=2
export MEM_FRACTION=0.85

IMAGE="docker://lmsysorg/sglang:kimi-k3"
RUN_SCRIPT=/home/ubuntu/workspace/kimi-k3-deploy/sglang/run_node_sglang.sh

HEAD_NODE=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
export NODE0_IP=$(srun --nodes=1 --ntasks=1 -w "$HEAD_NODE" hostname -i | awk '{print $1}')

echo "============================================"
echo "Kimi-K3 SGLang 2-node TP16+EP16 (p5e)"
echo "Nodes: $SLURM_JOB_NODELIST"
echo "Head:  $HEAD_NODE ($NODE0_IP)"
echo "API:   http://${NODE0_IP}:${HOST_PORT}/v1"
echo "============================================"

srun --container-image="$IMAGE" \
     --container-mounts=/home:/home \
     --export=ALL \
     bash "$RUN_SCRIPT" \
     2>&1 | tee /home/ubuntu/workspace/kimi-k3-deploy/logs/sgl_server_${SLURM_JOB_ID}.log

Hi @local0state , I see that you closed the discussion. Have you found a solution to get Kimi K3 working with vLLM with 2x8 H200 ?

@mateo-hug I don't think we have a solution at our user side, have to wait for vLLM to fix the memory leakage. Haven't tested afterwards, don't know if it's fixed or not yet.

Fixed https://github.com/vllm-project/vllm/pull/51131