|
|
--- |
|
|
license: mit |
|
|
library_name: distributed-llama |
|
|
tags: |
|
|
- distributed-inference |
|
|
- text-generation |
|
|
--- |
|
|
|
|
|
This is converted [DeepSeek R1 Distill 8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B) model to Distributed Llama format. The model is quantized to Q40. |
|
|
|
|
|
## 🚀 How to Run? |
|
|
|
|
|
* ⏬ Download this repository. |
|
|
* ⏬ Download Distributed Llama repository. |
|
|
* 🔨 Build Distributed Llama: |
|
|
|
|
|
``` |
|
|
make dllama |
|
|
``` |
|
|
|
|
|
## 🚀 Run Distributed Llama: |
|
|
|
|
|
``` |
|
|
./dllama chat --model dllama_model_deepseek-r1-distill-llama-8b_q40.m --tokenizer dllama_tokenizer_deepseek-r1-distill-llama-8b.t --buffer-float-type q80 --nthreads 4 --max-seq-len 8192 |
|
|
``` |
|
|
|
|
|
## 🎩 License |
|
|
|
|
|
[MIT License](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/LICENSE) |
