Update README.md
#1
by zheyishine - opened
README.md
CHANGED
|
@@ -76,21 +76,25 @@ uv venv ~/my_ling_env
|
|
| 76 |
source ~/my_ling_env/bin/activate
|
| 77 |
|
| 78 |
git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git
|
| 79 |
-
|
|
|
|
| 80 |
|
| 81 |
pip install --upgrade pip
|
|
|
|
| 82 |
pip install -e "python"
|
| 83 |
```
|
| 84 |
|
| 85 |
#### Run Inference
|
| 86 |
-
Here is
|
| 87 |
|
| 88 |
**Server**
|
| 89 |
|
| 90 |
-
|
| 91 |
|
| 92 |
```bash
|
| 93 |
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
|
|
|
|
|
|
|
| 94 |
python -m sglang.launch_server \
|
| 95 |
--model-path $MODEL_PATH \
|
| 96 |
--dist-init-addr $MASTER_IP:2345 \
|
|
@@ -100,7 +104,6 @@ python -m sglang.launch_server \
|
|
| 100 |
--max-running-requests 64 \
|
| 101 |
--tp-size 4 \
|
| 102 |
--chunked-prefill-size 8192 \
|
| 103 |
-
--allow-auto-output-truncate \
|
| 104 |
--tool-call-parser ling3 \
|
| 105 |
--reasoning-parser ling3 \
|
| 106 |
--context-length 262144 \
|
|
@@ -112,7 +115,7 @@ python -m sglang.launch_server \
|
|
| 112 |
|
| 113 |
**Client**
|
| 114 |
|
| 115 |
-
|
| 116 |
|
| 117 |
```bash
|
| 118 |
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
|
@@ -148,7 +151,7 @@ Here is the example to run Ling-3.0-flash with 4 GPUs, where the server port is
|
|
| 148 |
|
| 149 |
**Server**
|
| 150 |
|
| 151 |
-
|
| 152 |
|
| 153 |
```bash
|
| 154 |
vllm serve "$MODEL_PATH" \
|
|
@@ -167,7 +170,7 @@ vllm serve "$MODEL_PATH" \
|
|
| 167 |
|
| 168 |
**Client**
|
| 169 |
|
| 170 |
-
|
| 171 |
|
| 172 |
```bash
|
| 173 |
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
|
|
|
| 76 |
source ~/my_ling_env/bin/activate
|
| 77 |
|
| 78 |
git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git
|
| 79 |
+
|
| 80 |
+
cd sglang_ling_v3
|
| 81 |
|
| 82 |
pip install --upgrade pip
|
| 83 |
+
|
| 84 |
pip install -e "python"
|
| 85 |
```
|
| 86 |
|
| 87 |
#### Run Inference
|
| 88 |
+
Here is an example to run Ling-3.0-flash with 4 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
|
| 89 |
|
| 90 |
**Server**
|
| 91 |
|
| 92 |
+
Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-algorithm NEXTN) for lower latency.
|
| 93 |
|
| 94 |
```bash
|
| 95 |
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
|
| 96 |
+
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
|
| 97 |
+
export SGLANG_ENABLE_SPEC_V2=1
|
| 98 |
python -m sglang.launch_server \
|
| 99 |
--model-path $MODEL_PATH \
|
| 100 |
--dist-init-addr $MASTER_IP:2345 \
|
|
|
|
| 104 |
--max-running-requests 64 \
|
| 105 |
--tp-size 4 \
|
| 106 |
--chunked-prefill-size 8192 \
|
|
|
|
| 107 |
--tool-call-parser ling3 \
|
| 108 |
--reasoning-parser ling3 \
|
| 109 |
--context-length 262144 \
|
|
|
|
| 115 |
|
| 116 |
**Client**
|
| 117 |
|
| 118 |
+
We recommend using the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`, and enabling `enable_thinking` for better performance.
|
| 119 |
|
| 120 |
```bash
|
| 121 |
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
|
|
|
| 151 |
|
| 152 |
**Server**
|
| 153 |
|
| 154 |
+
Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-config) for lower latency.
|
| 155 |
|
| 156 |
```bash
|
| 157 |
vllm serve "$MODEL_PATH" \
|
|
|
|
| 170 |
|
| 171 |
**Client**
|
| 172 |
|
| 173 |
+
We recommend using the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`, and enabling `enable_thinking` for better performance.
|
| 174 |
|
| 175 |
```bash
|
| 176 |
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|