Update README.md
#3
by Jiminator - opened
README.md
CHANGED
|
@@ -104,8 +104,22 @@ vllm serve \
|
|
| 104 |
|
| 105 |
#### SGLang
|
| 106 |
|
| 107 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
#### TRT-LLM
|
| 110 |
|
| 111 |
Laguna is supported in TensorRT-LLM thanks to the team at NVIDIA ([NVIDIA/TensorRT-LLM#13559](https://github.com/NVIDIA/TensorRT-LLM/pull/13559), with partial-RoPE fusion in [#15110](https://github.com/NVIDIA/TensorRT-LLM/pull/15110)). The full recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so no extra flags are required.
|
|
|
|
| 104 |
|
| 105 |
#### SGLang
|
| 106 |
|
| 107 |
+
The full SGLang recipe is on the [SGLang Cookbook](https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-M.1). Quantization is detected automatically, so no extra flags are required.
|
| 108 |
+
```shell
|
| 109 |
+
git clone https://github.com/sgl-project/sglang.git
|
| 110 |
+
cd sglang
|
| 111 |
+
pip install -e "python[all]"
|
| 112 |
|
| 113 |
+
sglang serve \
|
| 114 |
+
--model-path poolside/Laguna-M.1-FP8 \
|
| 115 |
+
--trust-remote-code \
|
| 116 |
+
--reasoning-parser poolside_v1 \
|
| 117 |
+
--tool-call-parser poolside_v1 \
|
| 118 |
+
--tp 8 \
|
| 119 |
+
--host 0.0.0.0 \
|
| 120 |
+
--port 30000 \
|
| 121 |
+
--fp8-gemm-backend triton # Needed for SM100
|
| 122 |
+
```
|
| 123 |
#### TRT-LLM
|
| 124 |
|
| 125 |
Laguna is supported in TensorRT-LLM thanks to the team at NVIDIA ([NVIDIA/TensorRT-LLM#13559](https://github.com/NVIDIA/TensorRT-LLM/pull/13559), with partial-RoPE fusion in [#15110](https://github.com/NVIDIA/TensorRT-LLM/pull/15110)). The full recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so no extra flags are required.
|