Update README.md
#5
by baranowskiadam - opened
README.md
CHANGED
|
@@ -88,11 +88,13 @@ Laguna M.1 has upstream support in vLLM, SGLang, and TRT-LLM thanks to the suppo
|
|
| 88 |
|
| 89 |
#### vLLM
|
| 90 |
|
| 91 |
-
The full vLLM recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-M.1-FP8` substituted for the model ID.
|
| 92 |
|
| 93 |
```shell
|
| 94 |
pip install 'vllm>=0.21.0'
|
| 95 |
|
|
|
|
|
|
|
| 96 |
vllm serve \
|
| 97 |
--model poolside/Laguna-M.1-FP8 \
|
| 98 |
--tool-call-parser poolside_v1 \
|
|
|
|
| 88 |
|
| 89 |
#### vLLM
|
| 90 |
|
| 91 |
+
The full vLLM recipe is on the main [Laguna M.1 model card](https://huggingface.co/poolside/Laguna-M.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-M.1-FP8` substituted for the model ID. Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` when serving with vLLM.
|
| 92 |
|
| 93 |
```shell
|
| 94 |
pip install 'vllm>=0.21.0'
|
| 95 |
|
| 96 |
+
export VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0
|
| 97 |
+
|
| 98 |
vllm serve \
|
| 99 |
--model poolside/Laguna-M.1-FP8 \
|
| 100 |
--tool-call-parser poolside_v1 \
|