File size: 1,606 Bytes
a31ef7f f347eff a31ef7f 6b8d347 a31ef7f 6b8d347 a31ef7f 6b8d347 a31ef7f 6b8d347 a31ef7f 6b8d347 a31ef7f abfb85a a31ef7f abfb85a a31ef7f 6b8d347 a31ef7f 6b8d347 abfb85a 6b8d347 6adaa67 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
---
license: apache-2.0
base_model: mistralai/Voxtral-Mini-3B-2507
tags:
- voxtral
- quantized
- mlx
- voxtral-mini-3b-2507
library_name: mlx
---
# Voxtral Mini 3B — 2507 — Quantized (MLX)
Public quantized weights based on MLX bf16 from `mlx-community/Voxtral-Mini-3B-2507-bf16`.
Upstream model: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507).
## Variants (quantization profiles)
- Q4: folder `mlx-q4/`
- Q5: folder `mlx-q5/`
- Q6: folder `mlx-q6/`
- Q8: folder `mlx-q8/`
Published variants appear as subfolders at the top of this repo when available.
## Quantization notes
- Only inference weights are quantized (Q4/Q5/Q6/Q8 as above).
- Embeddings are NOT quantized to preserve shape compatibility. Therefore, any "bits per weight" metric may exceed the nominal target (informational, not an error).
## Quickstart (MLX)
```python
from mlx_lm import load, generate
model, tokenizer = load("NeoRoth/voxtral-3b-quantized")
print(generate(model, tokenizer, "Hello!", max_tokens=64))
```
## Integrity (SHA256)
- Q4 `model-00001-of-00001.safetensors`:
- `eec98aef078b3db2c226943d38558d814b10ec387dc5359d333eeed4be5298d2`
- Q8 `model-00001-of-00001.safetensors`:
- `37999e4a9dda52a0aedb593636be6c12e69dd8b8457f15ce48134f88b1ccebd3`
## License
- Apache-2.0 (see `LICENSE.txt`).
## Credits
- Upstream model: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507)
- MLX bf16 base used for quantization: [`mlx-community/Voxtral-Mini-3B-2507-bf16`](https://huggingface.co/mlx-community/Voxtral-Mini-3B-2507-bf16)
|