Update README.md

README.md

@@ -171,6 +171,44 @@ curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
 
 More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
 
+### vLLM
+
+#### Environment Preparation
+
+```bash
+pip install vllm==0.11.0
+```
+
+#### Run Inference:
+
+Here is the example to deploy the model with multiple GPU nodes, where the master node IP is ${MASTER_IP}, server port is ${PORT} and the path of model is ${MODEL_PATH}:
+
+```bash
+# step 1. start ray on all nodes
+
+# step 2. start vllm server only on node 0:
+vllm serve $MODEL_PATH --port $PORT --served-model-name my_model --trust-remote-code --tensor-parallel-size 8 --pipeline-parallel-size 4 --gpu-memory-utilization 0.85
+
+
+# This is only an example, please adjust arguments according to your actual environment.
+```
+
+To handle long context in vLLM using YaRN, we need to follow these two steps:
+1. Add a `rope_scaling` field to the model's `config.json` file, for example:
+```json
+{
+  ...,
+  "rope_scaling": {
+    "factor": 4.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  }
+}
+```
+2. Use an additional parameter `--max-model-len` to specify the desired maximum context length when starting the vLLM service.
+
+For detailed guidance, please refer to the vLLM [`instructions`](https://docs.vllm.ai/en/latest/).
+
 ## Finetuning
 
 We recommend you use [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory) to [finetune Ring](https://github.com/inclusionAI/Ring-V2/blob/main/docs/llamafactory_finetuning.md).