Quantization question
#1
by
Galathana - opened
I tried to quantize Deepseek-v3.2-exp using the same methods you outlined, but am ending up with 234 safetensor files and a model larger than the original 690gb, which is, you know, kind of the opposite of what I wanted. 😂 Any thoughts on what could be going wrong/ how to fix it? What was your exact mlx_lm.convert command?
Yeah, that's going the wrong direction. 😄
mlx_lm.convert --hf-path The-path-to-downloaded-model-or-on-🤗 --mlx-path DeepSeek-V3.2-mlx-5bit -q --q-bits 5