| | --- |
| | tags: |
| | - uqff |
| | - mistral.rs |
| | base_model: mistralai/Devstral-Small-2507 |
| | base_model_relation: quantized |
| | --- |
| | |
| | <!-- Autogenerated from user input. --> |
| |
|
| | # `mistralai/Devstral-Small-2507`, UQFF quantization |
| |
|
| |
|
| | Run with [mistral.rs](https://github.com/EricLBuehler/mistral.rs). Documentation: [UQFF docs](https://github.com/EricLBuehler/mistral.rs/blob/master/docs/UQFF.md). |
| |
|
| | 1) **Flexible** 🌀: Multiple quantization formats in *one* file format with *one* framework to run them all. |
| | 2) **Reliable** 🔒: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1. |
| | 3) **Easy** 🤗: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file. |
| | 3) **Customizable** 🛠️: Make and publish your own UQFF files in minutes. |
| |
|
| | ## Examples |
| | |Quantization type(s)|Example| |
| | |--|--| |
| | |AFQ2|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-afq2-0.uqff`| |
| | |AFQ3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-afq3-0.uqff`| |
| | |AFQ4|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq4-0.uqff;mistraldevstral-small-2507-afq4-1.uqff"`| |
| | |AFQ6|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq6-0.uqff;mistraldevstral-small-2507-afq6-1.uqff"`| |
| | |AFQ8|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-afq8-0.uqff;mistraldevstral-small-2507-afq8-1.uqff;mistraldevstral-small-2507-afq8-2.uqff"`| |
| | |F8E4M3|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-f8e4m3-0.uqff;mistraldevstral-small-2507-f8e4m3-1.uqff;mistraldevstral-small-2507-f8e4m3-2.uqff"`| |
| | |Q2K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-q2k-0.uqff`| |
| | |Q3K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f mistraldevstral-small-2507-q3k-0.uqff`| |
| | |Q4K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q4k-0.uqff;mistraldevstral-small-2507-q4k-1.uqff"`| |
| | |Q5K|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q5k-0.uqff;mistraldevstral-small-2507-q5k-1.uqff"`| |
| | |Q8_0|`./mistralrs-server -i plain -m EricB/Devstral-Small-2507-UQFF -f "mistraldevstral-small-2507-q8_0-0.uqff;mistraldevstral-small-2507-q8_0-1.uqff;mistraldevstral-small-2507-q8_0-2.uqff"`| |
| |
|
