Instructions to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/deepseek-coder-6.7B-instruct-GGUF", dtype="auto")

llama-cpp-python

How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="TheBloke/deepseek-coder-6.7B-instruct-GGUF",
	filename="deepseek-coder-6.7b-instruct.Q2_K.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with Ollama:
```
ollama run hf.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
```

Unsloth Studio

How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheBloke/deepseek-coder-6.7B-instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheBloke/deepseek-coder-6.7B-instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for TheBloke/deepseek-coder-6.7B-instruct-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/deepseek-coder-6.7B-instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/deepseek-coder-6.7B-instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.deepseek-coder-6.7B-instruct-GGUF-Q4_K_M

List all available models

lemonade list

This model cannot be used normally

by hyunfzen - opened Nov 5, 2023

Discussion

hyunfzen

Nov 5, 2023

llama.cpp error:
libc++abi: terminating due to uncaught exception of type std::out_of_range: unordered_map::at: key not found

TheBloke

Owner Nov 5, 2023

ah shit yeah I see the same. Investigating

hyunfzen

Nov 5, 2023

I use it in web-ui, and I got

Hope that it is useful.

mufeed

Nov 5, 2023

I use it in web-ui, and I got

Hope that it is useful.

Same here too...

TheBloke

Owner Nov 5, 2023

•

edited Nov 5, 2023

I'm remaking all the GGUFs using an updated llama.cpp convert.py which I hope will solve the crashing issue in llama.cpp with the 6.7B model. At least it hasn't crashed for me yet.

Regarding the "byte not found in vocab", I think that's an issue in the client used by WebUI, and is not something I can solve. I can only confirm the models work with llama.cpp. After that it requires the developers of the various clients to make sure they're up-to-date with llama.cpp.

I think in that case it's because llama-cpp-python, which textgen-webui uses, has not been updated to support BPE vocab, which was recently worked on in llama.cpp.

deleted

Nov 5, 2023

will be really nice once all these libraries and formats stabilize. moving targets suck.

TheBloke

Owner Nov 5, 2023

There was also an issue with the EOS token on all the Instruct models. The source models had it set incorrectly, so the GGUFs would not stop generating.

I've reported that to DeepSeek, they've fixed it, and I am now re-re-re-making the three Instruct model GGUFs, which are in the process of uploading now.

jmp470

Nov 5, 2023

Thank you for doing this! I'm excited to try deepseek.

gmacgmac

Nov 6, 2023

•

edited Nov 6, 2023

I have this model deepseek-coder-6.7b-instruct.Q5_K_S.gguf working in TG WebUI on Mac M1 Silicon on CPU
I don't get the "byte not found in vocab" which is very curious
However I can't get it working on Mac M2 Max on CPU and get "byte not found in vocab"- strange

I did however have to change n_ctx = 16380 on M1 Mac for it to work
Not sure why that is but going to explore

supercharge19

Nov 7, 2023

I have this model deepseek-coder-6.7b-instruct.Q5_K_S.gguf working in TG WebUI on Mac M1 Silicon on CPU
I don't get the "byte not found in vocab" which is very curious
However I can't get it working on Mac M2 Max on CPU and get "byte not found in vocab"- strange

I did however have to change n_ctx = 16380 on M1 Mac for it to work
Not sure why that is but going to explore

any update?

gmacgmac

Nov 7, 2023

It appears to be my version of Llama.cpp
But even when I install that that module version it doesn't work in another environment so there's something about the way i built this one!!
I copied this module in and then I don't get the errors! any ideas how else to confirm?

llama-cpp-python 0.2.13 pypi_0 pypi

% pip show llama-cpp-python
Name: llama_cpp_python
Version: 0.2.13
Summary: Python bindings for the llama.cpp library
Home-page:
Author:
Author-email: Andrei Betlen abetlen@gmail.com
License: MIT
Location: /Users/gm/miniconda3/envs/textgen/lib/python3.11/site-packages
Requires: diskcache, numpy, typing-extensions

gmacgmac

Nov 7, 2023

This works for me on mac but in CPU mode only "n-gpu-layers=0"

pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python==0.2.13 --no-cache-dir
pip install 'llama-cpp-python[server]==0.2.13'

TheBloke

Owner Nov 7, 2023

Thanks for reporting. Looks like llama-cpp-python updated yesterday and now how has support for models like this, with BPE vocab

deleted

Nov 7, 2023

Sometimes patience is a virtue :)

supercharge19

Nov 7, 2023

sometimes llama cpp gets broken (even latest update a few days ago was broken)

mufeed

Nov 7, 2023

•

edited Nov 7, 2023

Upgrade llama_cpp_python to 0.2.14, and now it is working with CPU in Oobabooga on Linux:

./cmd_linux.sh
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

deleted

Nov 7, 2023

•

edited Nov 7, 2023

that also will work for CPU only. At least it did for me . I just did the force upgrade to 0.2.14 ( after i stopped my service , started conda env. manually.. bla bla bla.... my setup is a bit different )

....or at least it did make it load, still not tested how well it works. Just it loaded and spit out some text so i know it "worked"

JonoMcCarthy

Nov 8, 2023

•

edited Nov 8, 2023

THanks Mr Bloke for all your hard work:
Just tested with ctransformers
ERROR: byte not found in vocab: '

code I used
llm = AutoModelForCausalLM.from_pretrained(local_model, model_file="deepseek-coder-6.7b-instruct.Q4_K_M.gguf", model_type="deepseek", gpu_layers=50, local_files_only=True)

any ideas or tips to get it working?

supercharge19

Nov 10, 2023

ctransformers is not good at all, for some model it breaks and for most outputs are so weird and just not acceptable, try llama_cpp_python

rozek

Nov 24, 2023

I just downloaded "deepseek-coder-6.7b-instruct.Q5_K_M.gguf" and tested it with the current version of llama.cpp (the C/C++ variant) - and it works like a charm

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment