Serve Kimi-K3 on your own GPUs
For anyone trying to self-host K3: the weights are ~1.4 TB even in native MXFP4, which just barely fits on a single top-end 8-GPU node and leaves nothing for the 1M-token KV cache or throughput headroom. So it's multi-node, and in practice multi-cluster, since not many people have that much contiguous capacity sitting in one place.
That's the specific problem SkyPilot Endpoints targets. You describe the endpoint in one YAML and it stands up vLLM plus the autoscaler, gateway, cache-aware routing, and metrics across however many Kubernetes clusters you give it, behind one URL. Clusters are pooled rather than treated separately, so replicas go wherever there's capacity and get rebuilt on a healthy cluster if one dies, same URL.
Except the weights are not ~1.4 TB, as can be seen from a quick look at this repo...