CatherineZY commited on
Commit
7d53578
·
verified ·
1 Parent(s): 2d2391e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -8
README.md CHANGED
@@ -46,7 +46,10 @@ The following table presents representative benchmarks for Ling-3.0-tiny:
46
 
47
  # Quickstart
48
  ## SGLang
49
- The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake) includes a live command generator and verified configurations:
 
 
 
50
 
51
  ### Install SGLang
52
  Use the pre-built image that tracks the Ling-3.0 runtime:
@@ -56,7 +59,7 @@ docker pull lmsysorg/sglang:dev-Ling-3.0-tiny
56
  ```
57
 
58
  ### Run Inference
59
- We recommend the low-latency recipe (built-in MTP / NEXTN, 256K YaRN context) on 1× 141GB-class GPU (H20-3e) or a 1-GPU Blackwell node:
60
 
61
  **Server**
62
 
@@ -79,7 +82,7 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
79
 
80
  **Client**
81
 
82
- The chat template and the `ling3` reasoning parser enable thinking mode by default. To disable it for an individual request, set `"chat_template_kwargs": {"enable_thinking": false}`. We recommend the sampling parameters `temperature=1.0`, `top_p=0.95`, and `top_k=20`.
83
 
84
  ```bash
85
  curl -s http://localhost:30000/v1/chat/completions \
@@ -92,6 +95,7 @@ curl -s http://localhost:30000/v1/chat/completions \
92
  "top_p": 0.95
93
  }'
94
  ```
 
95
 
96
  ## vLLM
97
  ### Install our vLLM
@@ -110,7 +114,7 @@ VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
110
  ```
111
 
112
  ### Run Inference
113
- The following example runs Ling-3.0-tiny on 1 GPU, with the server listening on port `${PORT}`:
114
 
115
  **Server**
116
 
@@ -166,11 +170,11 @@ cmake --build build --parallel 8
166
  >
167
 
168
  ### Import Model
169
- Replace `/absolute/path/to/fp8_weights` with the absolute path to the FP8 model weights directory. The imported model will be named `ling-tiny-fp8`:
170
 
171
  ```bash
172
- printf 'FROM /absolute/path/to/fp8_weights\n' > /tmp/Modelfile.ling
173
- ./ollama create ling-tiny-fp8 --experimental -f /tmp/Modelfile.ling
174
  ```
175
 
176
  ### Start Service
@@ -184,7 +188,7 @@ OLLAMA_CONTEXT_LENGTH=8192 ./ollama serve
184
  ### Call API
185
  ```bash
186
  curl -sS http://127.0.0.1:11434/api/generate -d '{
187
- "model": "ling-tiny-fp8",
188
  "prompt": "<role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>Calculate 17 × 23 and output only the number.<|role_end|><role>ASSISTANT</role>\n<think>",
189
  "raw": true,
190
  "think": true,
 
46
 
47
  # Quickstart
48
  ## SGLang
49
+ The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake), with a live command generator and verified configurations, lives in the SGLang cookbook:
50
+
51
+ **Cookbook:** https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny
52
+
53
 
54
  ### Install SGLang
55
  Use the pre-built image that tracks the Ling-3.0 runtime:
 
59
  ```
60
 
61
  ### Run Inference
62
+ Recommended low-latency recipe (built-in MTP / NEXTN, 256K YaRN context) on 1× 141GB-class GPU (H20-3e) or a 1-GPU Blackwell node:
63
 
64
  **Server**
65
 
 
82
 
83
  **Client**
84
 
85
+ Thinking is enabled by default by both the chat template and the `ling3` reasoning parser. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. We recommend the sampling parameters `temperature=1.0`, `top_p=0.95`, and `top_k=20`.
86
 
87
  ```bash
88
  curl -s http://localhost:30000/v1/chat/completions \
 
95
  "top_p": 0.95
96
  }'
97
  ```
98
+ For `--reasoning-parser ling3` / `--tool-call-parser ling3`, the HiCache + Mooncake L3 setup, and GSM8K / bench_serving reproduction commands, see the cookbook page linked above.
99
 
100
  ## vLLM
101
  ### Install our vLLM
 
114
  ```
115
 
116
  ### Run Inference
117
+ Here is the example to run Ling-3.0-tiny with 1 GPU, where the server port is `${PORT}`:
118
 
119
  **Server**
120
 
 
170
  >
171
 
172
  ### Import Model
173
+ Replace `/absolute/path/to/bf16_weights` with the absolute path to the BF16 model weights directory. The imported model will be named `ling-tiny-bf16`:
174
 
175
  ```bash
176
+ printf 'FROM /absolute/path/to/bf16_weights\n' > /tmp/Modelfile.ling
177
+ ./ollama create ling-tiny-bf16 --experimental -f /tmp/Modelfile.ling
178
  ```
179
 
180
  ### Start Service
 
188
  ### Call API
189
  ```bash
190
  curl -sS http://127.0.0.1:11434/api/generate -d '{
191
+ "model": "ling-tiny-bf16",
192
  "prompt": "<role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>Calculate 17 × 23 and output only the number.<|role_end|><role>ASSISTANT</role>\n<think>",
193
  "raw": true,
194
  "think": true,