Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -108,7 +108,7 @@ Laguna XS 2.1-FP8 is supported in vLLM, SGLang and Transformers, and TRT-LLM tha
108
  #### vLLM
109
 
110
  The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. No extra flags required.
111
-
112
  > [!NOTE]
113
  > The FP8-quantized KV cache requires vLLM >= 0.22.0. Earlier versions produce scrambled output on non-Hopper GPUs because of a per-layer attention-head count bug, fixed in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650). On older vLLM, disable the FP8 KV cache by adding `--kv-cache-dtype-skip-layers $(seq 0 39)`.
114
 
 
108
  #### vLLM
109
 
110
  The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. No extra flags required.
111
+ The full vLLM recipe is on the main [Laguna XS 2.1 model card](https://huggingface.co/poolside/Laguna-XS-2.1) and on the [vLLM recipes page](https://recipes.vllm.ai/poolside/Laguna-XS-2.1). Quantization is detected automatically from `quantization_config` in this checkpoint, so the same command works with `poolside/Laguna-XS-2.1-FP8` substituted for the model ID. Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` when serving with vLLM.
112
  > [!NOTE]
113
  > The FP8-quantized KV cache requires vLLM >= 0.22.0. Earlier versions produce scrambled output on non-Hopper GPUs because of a per-layer attention-head count bug, fixed in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650). On older vLLM, disable the FP8 KV cache by adding `--kv-cache-dtype-skip-layers $(seq 0 39)`.
114