Update SGLang quickstart to the official SGLang cookbook recipe

README.md

@@ -76,71 +76,62 @@ We evaluate the quantized models using several datasets. The FP8 quantized model
 
 
 ## Quickstart
-### SGlang
-#### Install our SGLang
-```bash
-pip install uv
+### SGLang
 
-uv venv ~/my_ling_env
-
-source ~/my_ling_env/bin/activate
+The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake), with a live command generator and verified configurations, lives in the SGLang cookbook:
 
-git clone -b ling_v3_support https://github.com/inclusionAI/sglang_ling_v3.git
+**Cookbook:** https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash
 
-cd sglang_ling_v3
+#### Install SGLang
 
-pip install --upgrade pip
+Use the pre-built image that tracks the Ling-3.0 runtime:
 
-pip install -e "python"
+```bash
+docker pull lmsysorg/sglang:dev-Ling-3.0-flash
 ```
 
 #### Run Inference
-Here is an example to run Ling-3.0-flash with 4 GPUs, where the master node IP is `${MASTER_IP}` and server port is `${PORT}`:
-
-**Server**
 
-Since the model is trained with MTP, we recommend enabling MTP during inference (i.e., --speculative-algorithm NEXTN) for lower latency.
+Recommended low-latency recipe for the FP8 checkpoint (built-in MTP / NEXTN, 256K YaRN context) on 4× 141GB-class GPUs (H20-3e / H200) or 4-GPU Blackwell nodes (B200 / GB300). The checkpoint uses blockwise (128×128) E4M3 expert weights, so the FP8 recipes pair TP with expert parallelism; SGLang detects the quantization format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed:
 
 ```bash
-export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
-export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
-export SGLANG_ENABLE_SPEC_V2=1
-python -m sglang.launch_server \
-    --model-path $MODEL_PATH \
-    --dist-init-addr $MASTER_IP:2345 \
-    --port $PORT \
-    --nnodes 1 \
-    --mem-fraction-static 0.8 \
-    --max-running-requests 32 \
-    --tp-size 2 \
-    --chunked-prefill-size 8192 \
-    --allow-auto-truncate \
-    --tool-call-parser ling3 \
-    --reasoning-parser ling3 \
+docker run --rm --gpus all --ipc=host --shm-size 32g \
+  -p 30000:30000 \
+  -e HF_TOKEN=<your-hf-token> \
+  lmsysorg/sglang:dev-Ling-3.0-flash \
+  env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
+  python3 -m sglang.launch_server \
+    --model-path inclusionAI/Ling-3.0-flash-fp8 \
+    --tp 4 \
+    --ep-size 4 \
     --context-length 262144 \
     --speculative-algorithm NEXTN \
-    --max-mamba-cache-size 160 \
-    --enable-fp32-lm-head \
-    --disable-shared-experts-fusion
+    --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
+    --mem-fraction-static 0.8 \
+    --host 0.0.0.0 \
+    --port 30000
 ```
 
+On H100 / H800 use `--tp 8 --ep-size 8` with the same flags; see the cookbook cell for your hardware.
+
 **Client**
 
-We recommend using the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`, and enabling `enable_thinking` for better performance.
+Thinking is enabled by default by both the chat template and the `ling3` reasoning parser. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. We recommend the sampling parameters `temperature=0.6`, `top_p=0.95`, and `top_k=20`.
 
 ```bash
-curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+curl -s http://localhost:30000/v1/chat/completions \
   -H "Content-Type: application/json" \
-  -d '{"model": "auto",
+  -d '{"model": "inclusionAI/Ling-3.0-flash-fp8",
        "messages": [{"role": "user", "content": "hello!"}],
-       "chat_template_kwargs": {"enable_thinking": true},
        "stream": true,
-       "temperature": 0.6, 
+       "temperature": 0.6,
        "top_k": 20,
        "top_p": 0.95
      }'
 ```
 
+For `--reasoning-parser ling3` / `--tool-call-parser ling3`, the HiCache + Mooncake L3 setup, and GSM8K / `bench_serving` reproduction commands, see the cookbook page linked above.
+
 ### vLLM
 #### Install our vLLM
 ```bash