Update README.md
Browse files
README.md
CHANGED
|
@@ -142,27 +142,25 @@ pip3 install -U sglang sgl-kernel
|
|
| 142 |
|
| 143 |
Both BF16 and FP8 models are supported by SGLang now. It depends on the dtype of the model in ${MODEL_PATH}.
|
| 144 |
|
| 145 |
-
Here is the example to run [Ring-1T](https://zenmux.ai/inclusionai/ring-1t?utm_source=hf_inclusionAI) with multiple GPU nodes, where the master node IP is ${MASTER_IP} and port is ${PORT}:
|
| 146 |
|
| 147 |
- Start server:
|
| 148 |
```bash
|
| 149 |
# Node 0:
|
| 150 |
-
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP
|
| 151 |
|
| 152 |
# Node 1:
|
| 153 |
-
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP
|
| 154 |
|
| 155 |
# Node 2:
|
| 156 |
-
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP
|
| 157 |
|
| 158 |
# Node 3:
|
| 159 |
-
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP
|
| 160 |
|
| 161 |
# This is only an example. Please adjust arguments according to your actual environment.
|
| 162 |
```
|
| 163 |
|
| 164 |
-
MTP is supported for the base model, but not yet for the chat model. You can add parameter `--speculative-algorithm NEXTN` to the start command.
|
| 165 |
-
|
| 166 |
- Client:
|
| 167 |
|
| 168 |
```shell
|
|
|
|
| 142 |
|
| 143 |
Both BF16 and FP8 models are supported by SGLang now. It depends on the dtype of the model in ${MODEL_PATH}.
|
| 144 |
|
| 145 |
+
Here is the example to run [Ring-1T](https://zenmux.ai/inclusionai/ring-1t?utm_source=hf_inclusionAI) with multiple GPU nodes, where the master node IP is ${MASTER_IP} and server port is ${PORT}:
|
| 146 |
|
| 147 |
- Start server:
|
| 148 |
```bash
|
| 149 |
# Node 0:
|
| 150 |
+
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 0
|
| 151 |
|
| 152 |
# Node 1:
|
| 153 |
+
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 1
|
| 154 |
|
| 155 |
# Node 2:
|
| 156 |
+
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 2
|
| 157 |
|
| 158 |
# Node 3:
|
| 159 |
+
python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 3
|
| 160 |
|
| 161 |
# This is only an example. Please adjust arguments according to your actual environment.
|
| 162 |
```
|
| 163 |
|
|
|
|
|
|
|
| 164 |
- Client:
|
| 165 |
|
| 166 |
```shell
|