Update README.md

#1
by zheyishine - opened
Files changed (1) hide show
  1. README.md +10 -7
README.md CHANGED
@@ -76,21 +76,25 @@ uv venv ~/my_ling_env
76
  source ~/my_ling_env/bin/activate
77
 
78
  git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git
79
- cd sglang
 
80
 
81
  pip install --upgrade pip
 
82
  pip install -e "python"
83
  ```
84
 
85
  #### Run Inference
86
- Here is the example to run Ling-3.0-flash with 4 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
87
 
88
  **Server**
89
 
90
- As our model is trained with MTP, we recommend to turn on MTP (i.e., --speculative-algorithm NEXTN) in inference for lower latency.
91
 
92
  ```bash
93
  export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
 
 
94
  python -m sglang.launch_server \
95
  --model-path $MODEL_PATH \
96
  --dist-init-addr $MASTER_IP:2345 \
@@ -100,7 +104,6 @@ python -m sglang.launch_server \
100
  --max-running-requests 64 \
101
  --tp-size 4 \
102
  --chunked-prefill-size 8192 \
103
- --allow-auto-output-truncate \
104
  --tool-call-parser ling3 \
105
  --reasoning-parser ling3 \
106
  --context-length 262144 \
@@ -112,7 +115,7 @@ python -m sglang.launch_server \
112
 
113
  **Client**
114
 
115
- It is recommended to use the sampling params (<font style="color:rgb(16, 18, 20);">`temperature=0.6, top_p=0.95, top_k=20`</font>) and turn on `enable_thinking`for better performance.
116
 
117
  ```bash
118
  curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
@@ -148,7 +151,7 @@ Here is the example to run Ling-3.0-flash with 4 GPUs, where the server port is
148
 
149
  **Server**
150
 
151
- As our model is trained with MTP, we recommend to turn on MTP (i.e., --speculative-config) in inference for lower latency.
152
 
153
  ```bash
154
  vllm serve "$MODEL_PATH" \
@@ -167,7 +170,7 @@ vllm serve "$MODEL_PATH" \
167
 
168
  **Client**
169
 
170
- It is recommended to use the sampling params (<font style="color:rgb(16, 18, 20);">`temperature=0.6, top_p=0.95, top_k=20`</font>) and turn on `enable_thinking`for better performance.
171
 
172
  ```bash
173
  curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
 
76
  source ~/my_ling_env/bin/activate
77
 
78
  git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git
79
+
80
+ cd sglang_ling_v3
81
 
82
  pip install --upgrade pip
83
+
84
  pip install -e "python"
85
  ```
86
 
87
  #### Run Inference
88
+ Here is an example to run Ling-3.0-flash with 4 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
89
 
90
  **Server**
91
 
92
+ Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-algorithm NEXTN) for lower latency.
93
 
94
  ```bash
95
  export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
96
+ export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
97
+ export SGLANG_ENABLE_SPEC_V2=1
98
  python -m sglang.launch_server \
99
  --model-path $MODEL_PATH \
100
  --dist-init-addr $MASTER_IP:2345 \
 
104
  --max-running-requests 64 \
105
  --tp-size 4 \
106
  --chunked-prefill-size 8192 \
 
107
  --tool-call-parser ling3 \
108
  --reasoning-parser ling3 \
109
  --context-length 262144 \
 
115
 
116
  **Client**
117
 
118
+ We recommend using the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`, and enabling `enable_thinking` for better performance.
119
 
120
  ```bash
121
  curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
 
151
 
152
  **Server**
153
 
154
+ Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-config) for lower latency.
155
 
156
  ```bash
157
  vllm serve "$MODEL_PATH" \
 
170
 
171
  **Client**
172
 
173
+ We recommend using the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`, and enabling `enable_thinking` for better performance.
174
 
175
  ```bash
176
  curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \