add serving instructions for SGLang and TRT-LLM

README.md

@@ -88,7 +88,7 @@ The integration of foundation and fine-tuned models into AI systems requires add
 
 
 ## Inference:
-**Engine:** vLLM <br>
+**Engine:** vLLM/SGLang/TensorRT-LLM <br>
 **Test Hardware:** B200 <br>
 
 ## Post Training Quantization
@@ -103,6 +103,14 @@ To serve this checkpoint with [vLLM](https://github.com/vllm-project/vllm), you
 python3 -m vllm.entrypoints.openai.api_server --model nvidia/Kimi-K2-Thinking-NVFP4 --trust-remote-code --tensor-parallel-size 4
 ```
 
+To serve this checkpoint with [SGLang](https://github.com/sgl-project/sglang), you can start the docker `lmsysorg/sglang:latest` and run the sample command below:
+
+```sh
+python3 -m sglang.launch_server --model-path /home/scratch.omniml_data_2/HF_model_hub/Kimi-K2-Thinking-NVFP4 --tp 4 --quantization modelopt_fp4 --trust-remote-code
+```
+
+To serve this checkpoint with [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), please follow instructions in [deployment-guide-for-kimi-k2-thinking-on-trtllm](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/deployment-guide/deployment-guide-for-kimi-k2-thinking-on-trtllm.md)
+
 
 ## Model Limitations:
 The base model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.