Pythia Deduped Series GGML
This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.
For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).
Last updated on 2023-05-25.
For other versions of the models, see here:
- GGMLv1 q4_3 (70M to 12B)
- GGMLv1 q5_0 / q5_1 / q8_0 (70M to 2.8B)
- GGMLv1 q4_0 / q4_2 (70M to 2.8B)
- GGMLv2 q4_0 / q5_1 (70M to 2.8B)
- GGMLv3 q4_0 / q5_1 (70M to 2.8B)
Description:
- The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.
RAM USAGE
| Model | RAM usage |
|---|---|
| Unloaded | 41.3 MiB |
| ggmlv3-pythia-70m-deduped-q4_0.bin | 95.5 MiB |
| ggmlv3-pythia-160m-deduped-q4_0.bin | 201.1 MiB |
| ggmlv3-pythia-410m-deduped-q4_0.bin | 415.1 MiB |
| ggmlv3-pythia-1b-deduped-q4_0.bin | 762.2 MiB |
| ggmlv3-pythia-1.4b-deduped-q4_0.bin | 1.0 GiB |
| ggmlv3-pythia-2.8b-deduped-q4_0.bin | 1.9 GiB |
| ggmlv3-pythia-70m-deduped-q5_1.bin | 108.7 MiB |
| ggmlv3-pythia-160m-deduped-q5_1.bin | 226.9 MiB |
| ggmlv3-pythia-410m-deduped-q5_1.bin | 494.0 MiB |
| ggmlv3-pythia-1b-deduped-q5_1.bin | 943.9 MiB |
| ggmlv3-pythia-1.4b-deduped-q5_1.bin | 1.3 GiB |
| ggmlv3-pythia-2.8b-deduped-q5_1.bin | 2.3 GiB |
Tested on KoboldCpp with OpenBLAS enabled.