| license: apache-2.0 | |
| base_model: bigcode/starcoder2-3b | |
| language: | |
| - en | |
| tags: | |
| - mlx | |
| - ffai | |
| - quantized | |
| - 4bit | |
| - affine | |
| # starcoder2-3b-4bit | |
| 4-bit affine quantization of [bigcode/starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b), produced with [FFAI](https://github.com/thewafflehaus/FFAI) 0.1.0's `ffai convert` (mlx-affine format, `group_size=64`). | |
| ## Conversion | |
| ```bash | |
| ffai convert bigcode/starcoder2-3b --bits 4 \ | |
| --upload-repo ekryski/starcoder2-3b-4bit | |
| ``` | |
| ## See also | |
| - [FFAI](https://github.com/thewafflehaus/FFAI) — fast Apple Silicon LLM inference. `Model.load("ekryski/starcoder2-3b-4bit")` runs this checkpoint end-to-end. | |
| - [FFAI quickstart](https://github.com/thewafflehaus/FFAI/blob/main/documentation/quickstart.md) | |
| - [FFAI quantization docs](https://github.com/thewafflehaus/FFAI/blob/main/documentation/quantization.md) |