Update README.md
Browse files
README.md
CHANGED
|
@@ -46,7 +46,10 @@ The following table presents representative benchmarks for Ling-3.0-tiny:
|
|
| 46 |
|
| 47 |
# Quickstart
|
| 48 |
## SGLang
|
| 49 |
-
The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake)
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
### Install SGLang
|
| 52 |
Use the pre-built image that tracks the Ling-3.0 runtime:
|
|
@@ -56,7 +59,7 @@ docker pull lmsysorg/sglang:dev-Ling-3.0-tiny
|
|
| 56 |
```
|
| 57 |
|
| 58 |
### Run Inference
|
| 59 |
-
|
| 60 |
|
| 61 |
**Server**
|
| 62 |
|
|
@@ -79,7 +82,7 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
|
|
| 79 |
|
| 80 |
**Client**
|
| 81 |
|
| 82 |
-
|
| 83 |
|
| 84 |
```bash
|
| 85 |
curl -s http://localhost:30000/v1/chat/completions \
|
|
@@ -92,6 +95,7 @@ curl -s http://localhost:30000/v1/chat/completions \
|
|
| 92 |
"top_p": 0.95
|
| 93 |
}'
|
| 94 |
```
|
|
|
|
| 95 |
|
| 96 |
## vLLM
|
| 97 |
### Install our vLLM
|
|
@@ -110,7 +114,7 @@ VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
|
|
| 110 |
```
|
| 111 |
|
| 112 |
### Run Inference
|
| 113 |
-
|
| 114 |
|
| 115 |
**Server**
|
| 116 |
|
|
@@ -166,11 +170,11 @@ cmake --build build --parallel 8
|
|
| 166 |
>
|
| 167 |
|
| 168 |
### Import Model
|
| 169 |
-
Replace `/absolute/path/to/
|
| 170 |
|
| 171 |
```bash
|
| 172 |
-
printf 'FROM /absolute/path/to/
|
| 173 |
-
./ollama create ling-tiny-
|
| 174 |
```
|
| 175 |
|
| 176 |
### Start Service
|
|
@@ -184,7 +188,7 @@ OLLAMA_CONTEXT_LENGTH=8192 ./ollama serve
|
|
| 184 |
### Call API
|
| 185 |
```bash
|
| 186 |
curl -sS http://127.0.0.1:11434/api/generate -d '{
|
| 187 |
-
"model": "ling-tiny-
|
| 188 |
"prompt": "<role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>Calculate 17 × 23 and output only the number.<|role_end|><role>ASSISTANT</role>\n<think>",
|
| 189 |
"raw": true,
|
| 190 |
"think": true,
|
|
|
|
| 46 |
|
| 47 |
# Quickstart
|
| 48 |
## SGLang
|
| 49 |
+
The hardware- and recipe-specific launch matrix (BF16/FP8 × Low-Latency / High-Throughput / HiCache + Mooncake), with a live command generator and verified configurations, lives in the SGLang cookbook:
|
| 50 |
+
|
| 51 |
+
**Cookbook:** https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny
|
| 52 |
+
|
| 53 |
|
| 54 |
### Install SGLang
|
| 55 |
Use the pre-built image that tracks the Ling-3.0 runtime:
|
|
|
|
| 59 |
```
|
| 60 |
|
| 61 |
### Run Inference
|
| 62 |
+
Recommended low-latency recipe (built-in MTP / NEXTN, 256K YaRN context) on 1× 141GB-class GPU (H20-3e) or a 1-GPU Blackwell node:
|
| 63 |
|
| 64 |
**Server**
|
| 65 |
|
|
|
|
| 82 |
|
| 83 |
**Client**
|
| 84 |
|
| 85 |
+
Thinking is enabled by default by both the chat template and the `ling3` reasoning parser. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. We recommend the sampling parameters `temperature=1.0`, `top_p=0.95`, and `top_k=20`.
|
| 86 |
|
| 87 |
```bash
|
| 88 |
curl -s http://localhost:30000/v1/chat/completions \
|
|
|
|
| 95 |
"top_p": 0.95
|
| 96 |
}'
|
| 97 |
```
|
| 98 |
+
For `--reasoning-parser ling3` / `--tool-call-parser ling3`, the HiCache + Mooncake L3 setup, and GSM8K / bench_serving reproduction commands, see the cookbook page linked above.
|
| 99 |
|
| 100 |
## vLLM
|
| 101 |
### Install our vLLM
|
|
|
|
| 114 |
```
|
| 115 |
|
| 116 |
### Run Inference
|
| 117 |
+
Here is the example to run Ling-3.0-tiny with 1 GPU, where the server port is `${PORT}`:
|
| 118 |
|
| 119 |
**Server**
|
| 120 |
|
|
|
|
| 170 |
>
|
| 171 |
|
| 172 |
### Import Model
|
| 173 |
+
Replace `/absolute/path/to/bf16_weights` with the absolute path to the BF16 model weights directory. The imported model will be named `ling-tiny-bf16`:
|
| 174 |
|
| 175 |
```bash
|
| 176 |
+
printf 'FROM /absolute/path/to/bf16_weights\n' > /tmp/Modelfile.ling
|
| 177 |
+
./ollama create ling-tiny-bf16 --experimental -f /tmp/Modelfile.ling
|
| 178 |
```
|
| 179 |
|
| 180 |
### Start Service
|
|
|
|
| 188 |
### Call API
|
| 189 |
```bash
|
| 190 |
curl -sS http://127.0.0.1:11434/api/generate -d '{
|
| 191 |
+
"model": "ling-tiny-bf16",
|
| 192 |
"prompt": "<role>SYSTEM</role>detailed thinking on<|role_end|><role>HUMAN</role>Calculate 17 × 23 and output only the number.<|role_end|><role>ASSISTANT</role>\n<think>",
|
| 193 |
"raw": true,
|
| 194 |
"think": true,
|