File size: 877 Bytes
f2f3940 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | ---
license: apache-2.0
base_model: bigcode/starcoder2-3b
language:
- en
tags:
- mlx
- ffai
- quantized
- 4bit
- affine
---
# starcoder2-3b-4bit
4-bit affine quantization of [bigcode/starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b), produced with [FFAI](https://github.com/thewafflehaus/FFAI) 0.1.0's `ffai convert` (mlx-affine format, `group_size=64`).
## Conversion
```bash
ffai convert bigcode/starcoder2-3b --bits 4 \
--upload-repo ekryski/starcoder2-3b-4bit
```
## See also
- [FFAI](https://github.com/thewafflehaus/FFAI) — fast Apple Silicon LLM inference. `Model.load("ekryski/starcoder2-3b-4bit")` runs this checkpoint end-to-end.
- [FFAI quickstart](https://github.com/thewafflehaus/FFAI/blob/main/documentation/quickstart.md)
- [FFAI quantization docs](https://github.com/thewafflehaus/FFAI/blob/main/documentation/quantization.md) |