Fixed 5am spelling mistakes
Browse files
README.md
CHANGED
|
@@ -4,8 +4,8 @@ base_model: [deepseek-ai/DeepSeek-Coder-V2-Instruct]
|
|
| 4 |
|
| 5 |
#### Custom quantizations of deepseek-coder-v2-instruct optimized for cpu inference.
|
| 6 |
|
| 7 |
-
###
|
| 8 |
-
### While it required custom code to make, it is
|
| 9 |
|
| 10 |
>[!TIP]
|
| 11 |
>The following 4bit version is the one I use myself, it gets 17tps on 64 arm cores.
|
|
@@ -45,8 +45,9 @@ aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00003-of-00004.gguf \
|
|
| 45 |
aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf \
|
| 46 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf
|
| 47 |
```
|
|
|
|
| 48 |
>[!TIP]
|
| 49 |
-
> ### And for downloading the Q8_0 version converted in the most lossless way possible from hf bf16
|
| 50 |
>
|
| 51 |
```verilog
|
| 52 |
|
|
@@ -69,6 +70,6 @@ aria2c -x 8 -o deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf \
|
|
| 69 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf
|
| 70 |
```
|
| 71 |
|
| 72 |
-
Enjoy and
|
| 73 |
|
| 74 |
-Nisten
|
|
|
|
| 4 |
|
| 5 |
#### Custom quantizations of deepseek-coder-v2-instruct optimized for cpu inference.
|
| 6 |
|
| 7 |
+
### This iq4xm one uses GGML TYPE IQ_4_XS 4bit in combination with q8_0 bit so it runs fast with minimal loss and takes advantage of int8 optimizations on most newer server cpus.
|
| 8 |
+
### While it required custom code to make, it is compatible with standard llama.cpp from github or just search nisten in lmstudio.
|
| 9 |
|
| 10 |
>[!TIP]
|
| 11 |
>The following 4bit version is the one I use myself, it gets 17tps on 64 arm cores.
|
|
|
|
| 45 |
aria2c -x 8 -o deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf \
|
| 46 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_iq4xm.gguf-00004-of-00004.gguf
|
| 47 |
```
|
| 48 |
+
|
| 49 |
>[!TIP]
|
| 50 |
+
> ### And for downloading the Q8_0 version converted in the most lossless way possible from hf bf16 download these:
|
| 51 |
>
|
| 52 |
```verilog
|
| 53 |
|
|
|
|
| 70 |
https://huggingface.co/nisten/deepseek-coder-v2-inst-cpu-optimized-gguf/resolve/main/deepseek_coder_v2_cpu_q8_0-00006-of-00006.gguf
|
| 71 |
```
|
| 72 |
|
| 73 |
+
Enjoy and remember to accelerate!
|
| 74 |
|
| 75 |
-Nisten
|