Request: Bonsai-27B – extreme low-bit quants (IQ1_M / IQ1_S) and/or pruned version
Hello mradermacher,
I would like to request ultra‑low‑bit quantizations for the Bonsai‑27B model family by PrismML:
Original model: https://lmstudio.ai/models/prism-ml/bonsai-27b
HF: https://huggingface.co/lmstudio-community/Bonsai-27B-GGUF and https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit
There are already 1‑bit GGUF releases (Q1_0) from PrismML, but they still weigh ~4.7 GB for the language part.
I am trying to run this model on extremely limited hardware (only 2 GB of RAM(VRAM) total). I know it's a long shot, but I was wondering if you could produce even more aggressive quantizations, such as:
IQ1_M (or IQ1_S)
Q1_K (if llama.cpp supports it)
Any other experimental 1‑bit or 2‑bit imatrix quants that might reduce the file size below 3 GB (ideally closer to 2 GB).
Additionally, if there is any possibility to apply structured pruning (like the original Bonsai pruning technique used to create the model) to further reduce the model size before quantization, that would be even better. I know this is far more complex and time‑consuming, but maybe you have tooling or experience with that? I'm open to any combination of pruning + quantization that could squeeze the model into ~2 GB.
I fully understand that quality will degrade significantly – I'm ready to accept that trade‑off for the sake of fitting into RAM.
If neither is feasible, could you please suggest alternative approaches (e.g., splitting, offloading, etc.) that might help?
Thank you for considering this request, and for all the great work you do for the community!
Best regards
To get it into 2gb, you would need to do structural pruning into 14b (in 1bit it will be 2gb), which might not work with llamacpp as it uses predefined architectures. Next thing it would need to be a real 1 bit quantization, which Im not sure if llamacpp can even do (last time I check a while ago, was ~1.6 bpw), you might want to do some quantization aware training with some sort of 1 bit layers yourself, without help of llamacpp (at least Im not aware of true 1bit support from llamacpp, but dont blindly trust my bad memory). So without some magic idk if you can run this thing on 2gb without swap space. You could use something like airllm (might be wrong name), which reads model on the flight, so memory requirement would be around the biggest layer memory + cache
But we, as team mradermacher, only provide ggufs, we dont prune or do magic. We might do some random stuff and experiments, but not putting it here