| | --- |
| | datasets: |
| | - HuggingFaceH4/ultrachat_200k |
| | base_model: |
| | - mistralai/Mistral-Small-24B-Instruct-2501 |
| | pipeline_tag: text-generation |
| | tags: |
| | - mistral |
| | - quantization |
| | - vllm |
| | - nvfp4 |
| | license: apache-2.0 |
| | --- |
| | # Model Card for ealexeev/Mistral-Small-24B-NVFP4 |
| |
|
| | ## Model Description |
| | This is a compressed version of Mistral Small 24B Instruct, quantized to **NVFP4** using `llm-compressor`. |
| |
|
| | **This model is optimized for NVIDIA Blackwell/Hopper GPUs (H100, B200) and vLLM.** |
| |
|
| | ## Benchmarks (Run on DGX Spark) |
| | | Metric | Base Model (FP16) | This Model (NVFP4) | Delta | |
| | | :--- | :--- | :--- | :--- | |
| | | **HellaSwag (Logic)** | 83.47% | 83.20% | -0.27% | |
| | | **IFEval (Strict)** | 71.46% | 70.50% | -0.96% | |
| | | **Throughput** | 712 tok/s | 1344 tok/s | **+1.88x** | |
| |
|
| | ## Usage |
| | **⚠️ This model requires vLLM. It will NOT work with GGUF/llama.cpp/Ollama.** |
| |
|
| | ### vLLM Command |
| | ```bash |
| | vllm serve ealexeev/Mistral-Small-24B-NVFP4 \ |
| | --tensor-parallel-size 1 \ |
| | --gpu-memory-utilization 0.8 \ |
| | --enforce-eager |
