TheBloke
/

WizardMath-7B-V1.0-GGML

Transformers

llama

Model card Files Files and versions

xet

Community

TheBloke commited on Aug 11, 2023

Commit

04f2eec

1 Parent(s): 6dc44af

Initial GGML model commit

Browse files

Files changed (1) hide show

README.md +21 -21

README.md CHANGED Viewed

@@ -3,7 +3,7 @@ inference: false
 license: other
 model_creator: WizardLM
 model_link: https://huggingface.co/WizardLM/WizardMath-7b-V1.0
-model_name: WizardMath 13B V1.0
 model_type: llama
 quantized_by: TheBloke
 ---
@@ -22,13 +22,13 @@ quantized_by: TheBloke
 </div>
 <!-- header end -->
-# WizardMath 13B V1.0 - GGML
 - Model creator: [WizardLM](https://huggingface.co/WizardLM)
-- Original model: [WizardMath 13B V1.0](https://huggingface.co/WizardLM/WizardMath-7b-V1.0)
 ## Description
-This repo contains GGML format model files for [WizardLM's WizardMath 13B V1.0](https://huggingface.co/WizardLM/WizardMath-7b-V1.0).
 GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
 * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most popular web UI. Supports NVidia CUDA GPU acceleration.
@@ -40,8 +40,8 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
 ## Repositories available
-* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GPTQ)
-* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML)
 * [WizardLM's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/WizardLM/WizardMath-7b-V1.0)
 ## Prompt template: Alpaca-CoT
@@ -84,20 +84,20 @@ Refer to the Provided Files table below to see what files use which methods, and
 | Name | Quant method | Bits | Size | Max RAM required | Use case |
 | ---- | ---- | ---- | ---- | ---- | ----- |
-| [wizardmath-7b-v1.0.ggmlv3.q2_K.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q2_K.bin) | q2_K | 2 | 3.05 GB| 5.55 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
-| [wizardmath-7b-v1.0.ggmlv3.q3_K_L.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_L.bin) | q3_K_L | 3 | 3.77 GB| 6.27 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
-| [wizardmath-7b-v1.0.ggmlv3.q3_K_M.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_M.bin) | q3_K_M | 3 | 3.45 GB| 5.95 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
-| [wizardmath-7b-v1.0.ggmlv3.q3_K_S.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_S.bin) | q3_K_S | 3 | 3.12 GB| 5.62 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
-| [wizardmath-7b-v1.0.ggmlv3.q4_0.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_0.bin) | q4_0 | 4 | 3.79 GB| 6.29 GB | Original quant method, 4-bit. |
-| [wizardmath-7b-v1.0.ggmlv3.q4_1.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_1.bin) | q4_1 | 4 | 4.21 GB| 6.71 GB | Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
-| [wizardmath-7b-v1.0.ggmlv3.q4_K_M.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_K_M.bin) | q4_K_M | 4 | 4.24 GB| 6.74 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
-| [wizardmath-7b-v1.0.ggmlv3.q4_K_S.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_K_S.bin) | q4_K_S | 4 | 3.98 GB| 6.48 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
-| [wizardmath-7b-v1.0.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_0.bin) | q5_0 | 5 | 4.63 GB| 7.13 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
-| [wizardmath-7b-v1.0.ggmlv3.q5_1.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_1.bin) | q5_1 | 5 | 5.06 GB| 7.56 GB | Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
-| [wizardmath-7b-v1.0.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 4.92 GB| 7.42 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
-| [wizardmath-7b-v1.0.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 4.79 GB| 7.29 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
-| [wizardmath-7b-v1.0.ggmlv3.q6_K.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q6_K.bin) | q6_K | 6 | 5.65 GB| 8.15 GB | New k-quant method. Uses GGML_TYPE_Q8_K for all tensors - 6-bit quantization |
-| [wizardmath-7b-v1.0.ggmlv3.q8_0.bin](https://huggingface.co/TheBloke/WizardMath-13B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q8_0.bin) | q8_0 | 8 | 7.16 GB| 9.66 GB | Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
 **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
@@ -151,7 +151,7 @@ Thank you to all my generous patrons and donaters!
 <!-- footer end -->
-# Original model card: WizardLM's WizardMath 13B V1.0

 license: other
 model_creator: WizardLM
 model_link: https://huggingface.co/WizardLM/WizardMath-7b-V1.0
+model_name: WizardMath 7B V1.0
 model_type: llama
 quantized_by: TheBloke
 ---
 </div>
 <!-- header end -->
+# WizardMath 7B V1.0 - GGML
 - Model creator: [WizardLM](https://huggingface.co/WizardLM)
+- Original model: [WizardMath 7B V1.0](https://huggingface.co/WizardLM/WizardMath-7b-V1.0)
 ## Description
+This repo contains GGML format model files for [WizardLM's WizardMath 7B V1.0](https://huggingface.co/WizardLM/WizardMath-7b-V1.0).
 GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
 * [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most popular web UI. Supports NVidia CUDA GPU acceleration.
 ## Repositories available
+* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GPTQ)
+* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML)
 * [WizardLM's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/WizardLM/WizardMath-7b-V1.0)
 ## Prompt template: Alpaca-CoT
 | Name | Quant method | Bits | Size | Max RAM required | Use case |
 | ---- | ---- | ---- | ---- | ---- | ----- |
+| [wizardmath-7b-v1.0.ggmlv3.q2_K.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q2_K.bin) | q2_K | 2 | 3.05 GB| 5.55 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
+| [wizardmath-7b-v1.0.ggmlv3.q3_K_L.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_L.bin) | q3_K_L | 3 | 3.77 GB| 6.27 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| [wizardmath-7b-v1.0.ggmlv3.q3_K_M.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_M.bin) | q3_K_M | 3 | 3.45 GB| 5.95 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| [wizardmath-7b-v1.0.ggmlv3.q3_K_S.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q3_K_S.bin) | q3_K_S | 3 | 3.12 GB| 5.62 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
+| [wizardmath-7b-v1.0.ggmlv3.q4_0.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_0.bin) | q4_0 | 4 | 3.79 GB| 6.29 GB | Original quant method, 4-bit. |
+| [wizardmath-7b-v1.0.ggmlv3.q4_1.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_1.bin) | q4_1 | 4 | 4.21 GB| 6.71 GB | Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+| [wizardmath-7b-v1.0.ggmlv3.q4_K_M.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_K_M.bin) | q4_K_M | 4 | 4.24 GB| 6.74 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
+| [wizardmath-7b-v1.0.ggmlv3.q4_K_S.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q4_K_S.bin) | q4_K_S | 4 | 3.98 GB| 6.48 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
+| [wizardmath-7b-v1.0.ggmlv3.q5_0.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_0.bin) | q5_0 | 5 | 4.63 GB| 7.13 GB | Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
+| [wizardmath-7b-v1.0.ggmlv3.q5_1.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_1.bin) | q5_1 | 5 | 5.06 GB| 7.56 GB | Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
+| [wizardmath-7b-v1.0.ggmlv3.q5_K_M.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_K_M.bin) | q5_K_M | 5 | 4.92 GB| 7.42 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
+| [wizardmath-7b-v1.0.ggmlv3.q5_K_S.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q5_K_S.bin) | q5_K_S | 5 | 4.79 GB| 7.29 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
+| [wizardmath-7b-v1.0.ggmlv3.q6_K.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q6_K.bin) | q6_K | 6 | 5.65 GB| 8.15 GB | New k-quant method. Uses GGML_TYPE_Q8_K for all tensors - 6-bit quantization |
+| [wizardmath-7b-v1.0.ggmlv3.q8_0.bin](https://huggingface.co/TheBloke/WizardMath-7B-V1.0-GGML/blob/main/wizardmath-7b-v1.0.ggmlv3.q8_0.bin) | q8_0 | 8 | 7.16 GB| 9.66 GB | Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
 **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
 <!-- footer end -->
+# Original model card: WizardLM's WizardMath 7B V1.0