Update README.md

README.md

@@ -209,72 +209,62 @@ print("\nGenerated Output:\n", decoded_output)
 ```
 
 ### Using vLLM (for optimized FP8 inference)
-> [!NOTE]
-> The following vLLM inference example is provided as a general guideline based on `llm-compressor`'s intended compatibility with vLLM for FP8 models. However, at the time of writing, this specific quantized model (`textgeflecht/Devstral-Small-2505-FP8-llmcompressor`) with its `MistralTokenizer` (using `tekken.json`) **has not been explicitly tested with vLLM by the author.**
-> 
-> Successfully running this model with vLLM might require:
-> * The latest version of vLLM with robust support for `compressed-tensors` format and custom tokenizers.
-> * Potential adjustments to tokenizer loading or prompt formatting to align with vLLM's expectations for models using `MistralTokenizer`.
-> * Consulting the [vLLM documentation](https://docs.vllm.ai/en/latest/) for the most up-to-date instructions on running quantized models and handling custom tokenizers.
-> 
-> Users are encouraged to experiment and consult vLLM resources if they encounter issues.
 
+This model, quantized to FP8 with `llm-compressor`, is designed for efficient inference with vLLM, especially on newer NVIDIA GPUs.
+
+**Prerequisites:**
+* A recent version of vLLM. The author's successful tests used a custom, very recent build of vLLM with specific patches for NVIDIA Blackwell FP8 support.
+* A compatible NVIDIA GPU (Ada Lovelace, Hopper, Blackwell, or newer architectures are recommended for FP8).
+* Docker and NVIDIA Container Toolkit installed.
+
+**Running with Docker (Recommended & Tested by Author):**
+
+The following Docker command starts a vLLM OpenAI-compatible server with this quantized model. This setup has been verified by the author to load the model successfully and serve requests.
+
+```bash
+# 1. Set your Hugging Face Token (optional, but recommended to avoid rate limits or for private models)
+# export HF_TOKEN="YOUR_HUGGINGFACE_ACCESS_TOKEN_HERE"
+
+# 2. Run the vLLM Docker container.
+# Replace 'vllm/vllm-openai:latest' with your specific vLLM image if using a custom build.
+# The 'latest' tag should pull a recent official build from vLLM.
+sudo docker run --gpus all \
+    -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
+    -p 8000:8000 \
+    -e HUGGING_FACE_HUB_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
+    vllm/vllm-openai:latest \
+    --model textgeflecht/Devstral-Small-2505-FP8-llmcompressor \
+    --tokenizer_mode mistral \
+    --load_format auto \
+    --max-model-len 2048 # Optional: Limit VRAM usage
+
+    # Optional: Add Mistral-specific tool usage flags if needed by your application.
+    # --tool-call-parser mistral \
+    # --enable-auto-tool-choice \
+    # Optional: Explicitly set tensor parallel size if you have multiple GPUs e.g. on 2 GPUs:
+    # --tensor-parallel-size 2
+```
+Key Command-Line Arguments Used:
 
-Ensure you have vllm installed (pip install vllm). 
-This requires a compatible GPU (NVIDIA Ada/Hopper/Blackwell or newer architecture).
-
-```python
-from vllm import LLM, SamplingParams
-
-# Define your model ID
-MODEL_REPO_ID = "textgeflecht/Devstral-Small-2505-FP8-llmcompressor"
+- `--model textgeflecht/Devstral-Small-2505-FP8-llmcompressor`: Specifies the quantized model from the Hugging Face Hub.
+- `--tokenizer_mode mistral`: Essential for vLLM to correctly use the MistralTokenizer with the `tekken.json` file from the repository.
+- `--load_format auto`: Allows vLLM to auto-detect the Hugging Face sharded safetensors format for weights. With this, vLLM successfully reads the `config.json` (which includes `quantization_config` with `quant_method: "compressed-tensors"`) and auto-detects the FP8 quantization scheme.
+- `--max-model-len 2048`: Limits the maximum sequence length (input + output tokens combined) to manage VRAM. Adjust this value based on your needs and available GPU memory.
+- The flags `--tool-call-parser mistral` and `--enable-auto-tool-choice` can be added if you intend to use Devstral's tool-calling capabilities.
 
-# Initialize vLLM with the model
-# vLLM should automatically detect the FP8 quantization if the format is compatible.
-# The 'quantization' parameter might be needed if auto-detection fails,
-# often models from llm-compressor are saved in 'compressed-tensors' format
-# which vLLM aims to support. You might need to specify tokenizer_mode='custom'
-# and point to the tekken.json or use a HuggingFace compatible tokenizer wrapper if needed.
-
-# For MistralTokenizer, vLLM might require manual handling or a specific setup.
-# If vLLM doesn't directly support MistralTokenizer from tekken.json,
-# you might need to use a HF AutoTokenizer equivalent IF one can be made compatible,
-# or wait for broader custom tokenizer support in vLLM.
-
-# Simplest case (assuming vLLM handles the compressed-tensors format and tokenizer):
-llm = LLM(model=MODEL_REPO_ID, trust_remote_code=True) 
-# If tekken.json is present, vLLM might use it if it knows the tokenizer type.
-# Or, if a compatible tokenizer_config.json can be created for MistralTokenizer.
-
-# For now, a more robust way with vLLM and custom tokenizers like Mistral's
-# might involve more direct specification or ensuring the saved format is fully
-# vLLM auto-detectable. If issues arise, consulting vLLM documentation for custom/FP8 models is key.
-
-# --- Placeholder for vLLM with MistralTokenizer ---
-# Direct vLLM usage with MistralTokenizer from a tekken.json might be tricky
-# as vLLM typically relies on Hugging Face AutoTokenizer.
-# One approach could be to use the HF AutoModelForCausalLM loading mechanism
-# and then pass it to vLLM if supported, or ensure the saved format is directly consumable.
-
-# Let's assume for now a standard prompt for demonstration if vLLM picks up a compatible tokenizer.
-# If not, you'd use MistralTokenizer to prepare the prompt as in the transformers example.
-
-prompts = [
-    "[INST] Write a python function that calculates the factorial of a number. [/INST]",
-    # Add more prompts if needed
-]
+Note on FP8 Support (especially for newer architectures like Blackwell):
+- vLLM's support for FP8, particularly on the newest GPU architectures like NVIDIA Blackwell, is an area of active development.
+- The successful tests for this model on Blackwell used a custom, very recent vLLM build with specific patches for Blackwell FP8 support.
+- While standard `vllm/vllm-openai:latest` images are updated regularly, cutting-edge hardware support and specific quantization schemes might take time to be fully integrated and stabilized in official releases.
+- If you encounter issues related to FP8 performance or compatibility on very new hardware with official vLLM builds, it's recommended to check the vLLM GitHub repository issues and discussions for the latest status, potential workarounds, or information on required builds.
 
-sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)
+Interacting with the Server:
+- Once the vLLM server is running, it exposes an OpenAI-compatible API.
+- You can interact with it using any OpenAI client library (like openai for Python) or tools like curl.
+- Endpoint for chat completions: `http://localhost:8000/v1/chat/completions`
+- Model name in requests: Use `textgeflecht/Devstral-Small-2505-FP8-llmcompressor`
 
-outputs = llm.generate(prompts, sampling_params)
-
-# Print the outputs
-for output in outputs:
-    prompt = output.prompt
-    generated_text = output.outputs[0].text
-    print(f"Prompt: {prompt!r}")
-    print(f"Generated text: {generated_text!r}\n---")
-```
+Refer to the Python requests example in the original https://huggingface.co/mistralai/Devstral-Small-2505 model card for client-side interaction, adjusting the URL and model name as needed.
 
-Original Model Card (mistralai/Devstral-Small-2505)
+### Original Model Card (mistralai/Devstral-Small-2505)
 For more details on the base model, please refer to the original model card: https://huggingface.co/mistralai/Devstral-Small-2505