Update README.md
#1
by yli-nexa4ai - opened
README.md
CHANGED
|
@@ -66,4 +66,61 @@ Get the latest version from the [official website](https://lmstudio.ai/).
|
|
| 66 |
1. In LM Studio's top panel, search for and select `NexaAIDev/DeepSeek-R1-Distill-Llama-8B-NexaQuant`.
|
| 67 |
2. Click `Download` (if not already downloaded) and wait for the model to load.
|
| 68 |
3. Once loaded, go to the chat window and start a conversation.
|
| 69 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
1. In LM Studio's top panel, search for and select `NexaAIDev/DeepSeek-R1-Distill-Llama-8B-NexaQuant`.
|
| 67 |
2. Click `Download` (if not already downloaded) and wait for the model to load.
|
| 68 |
3. Once loaded, go to the chat window and start a conversation.
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
## Example
|
| 72 |
+
|
| 73 |
+
On the left, we have an example of what LMStudio Q4_K_M responded. On the right is our NexaQuant version.
|
| 74 |
+
|
| 75 |
+
Prompt: A Common Investment Banking BrainTeaser Question
|
| 76 |
+
|
| 77 |
+
There is a 6x8 rectangular chocolate bar made up of small 1x1 bits. We want to break it into the 48 bits. We can break one piece of chocolate horizontally or vertically, but cannot break two pieces together! What is the minimum number of breaks required?
|
| 78 |
+
|
| 79 |
+
Right Answer: 47
|
| 80 |
+
|
| 81 |
+
<div align="center">
|
| 82 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/66abfd6f65beb23afa427d8a/ZS9e66t7OhBIno4eQ3OaX.png" width="80%" alt="Example" />
|
| 83 |
+
</div>
|
| 84 |
+
|
| 85 |
+
## Benchmarks
|
| 86 |
+
|
| 87 |
+
NexaQuant on Reasoning Benchmarks Compared to BF16 and LMStudio's Q4_K_M
|
| 88 |
+
|
| 89 |
+
**8B:**
|
| 90 |
+
|
| 91 |
+
<div align="center">
|
| 92 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/66abfd6f65beb23afa427d8a/4SSDIqxikgaDbx8V-Q6bf.png" width="80%" alt="Example" />
|
| 93 |
+
</div>
|
| 94 |
+
|
| 95 |
+
The general capacity has also greatly improved:
|
| 96 |
+
|
| 97 |
+
**8B:**
|
| 98 |
+
|
| 99 |
+
| Benchmark | Full 16-bit | llama.cpp (4-bit) | NexaQuant (4-bit)|
|
| 100 |
+
|----------------------------|------------|-------------------|-------------------|
|
| 101 |
+
| **HellaSwag** | 57.07 | 52.12 | 54.56 |
|
| 102 |
+
| **MMLU** | 55.59 | 52.82 | 54.94 |
|
| 103 |
+
| **Humanities** | 50.48 | 44.74 | 50.24 |
|
| 104 |
+
| **Social Sciences** | 65.32 | 65.62 | 64.74 |
|
| 105 |
+
| **STEM** | 47.73 | 52.50 | 46.75 |
|
| 106 |
+
| **ARC Easy** | 74.49 | 69.32 | 71.72 |
|
| 107 |
+
| **MathQA** | 35.34 | 30.00 | 32.46 |
|
| 108 |
+
| **PIQA** | 78.56 | 76.09 | 77.68 |
|
| 109 |
+
| **IFEval - Inst - Loose** | 44.24 | 44.95 | 42.45 |
|
| 110 |
+
| **IFEval - Inst - Strict** | 42.57 | 44.95 | 40.05 |
|
| 111 |
+
| **IFEval - Prom - Loose** | 30.31 | 25.74 | 28.47 |
|
| 112 |
+
| **IFEval - Prom - Strict** | 27.91 | 25.74 | 25.51 |
|
| 113 |
+
|
| 114 |
+
## What's next
|
| 115 |
+
|
| 116 |
+
1. Inference Nexa Quantized Deepseek-R1 distilled model on NPU.
|
| 117 |
+
|
| 118 |
+
2. This model is designed for complex problem-solving, which is why it has a longer thinking process. We understand this can be an issue in some cases, and we're actively working on improvements.
|
| 119 |
+
|
| 120 |
+
### Follow us
|
| 121 |
+
|
| 122 |
+
If you liked our work, feel free to ⭐Star [Nexa's GitHub Repo](https://github.com/NexaAI/nexa-sdk).
|
| 123 |
+
|
| 124 |
+
Interested in running DeepSeek R1 on your own devices with optimized CPU, GPU, and NPU acceleration or compressing your finetuned DeepSeek-Distill-R1? [Let’s chat!](https://nexa.ai/book-a-call)
|
| 125 |
+
|
| 126 |
+
[Blogs](https://nexa.ai/blogs/quantized-deepseek-r1-on-device) | [Discord](https://discord.gg/nexa-ai) | [X(Twitter)](https://x.com/nexa_ai)
|