Update README.md
#2
by baranowskiadam - opened
README.md
CHANGED
|
@@ -108,7 +108,7 @@ Laguna XS 2.1-FP8 is supported in vLLM, SGLang and Transformers, and TRT-LLM tha
|
|
| 108 |
#### vLLM
|
| 109 |
|
| 110 |
The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. No extra flags required.
|
| 111 |
-
|
| 112 |
> [!NOTE]
|
| 113 |
> The FP8-quantized KV cache requires vLLM >= 0.22.0. Earlier versions produce scrambled output on non-Hopper GPUs because of a per-layer attention-head count bug, fixed in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650). On older vLLM, disable the FP8 KV cache by adding `--kv-cache-dtype-skip-layers $(seq 0 39)`.
|
| 114 |
|
|
|
|
| 108 |
#### vLLM
|
| 109 |
|
| 110 |
The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. No extra flags required.
|
| 111 |
+
The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` when serving with vLLM.
|
| 112 |
> [!NOTE]
|
| 113 |
> The FP8-quantized KV cache requires vLLM >= 0.22.0. Earlier versions produce scrambled output on non-Hopper GPUs because of a per-layer attention-head count bug, fixed in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650). On older vLLM, disable the FP8 KV cache by adding `--kv-cache-dtype-skip-layers $(seq 0 39)`.
|
| 114 |
|