Instructions to use TheBloke/WizardCoder-15B-1.0-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/WizardCoder-15B-1.0-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/WizardCoder-15B-1.0-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/WizardCoder-15B-1.0-GPTQ")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/WizardCoder-15B-1.0-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/WizardCoder-15B-1.0-GPTQ

SGLang

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/WizardCoder-15B-1.0-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/WizardCoder-15B-1.0-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/WizardCoder-15B-1.0-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/WizardCoder-15B-1.0-GPTQ
```

keeps crashing when running on text-generation-webui

#13

by wesleysanjose - opened Jul 19, 2023

Discussion

wesleysanjose

Jul 19, 2023

•

edited Jul 19, 2023

tried to run this GPTQ using autoGPTQ in text-generation-webui, but it keeps crashing after interfering for seconds. i can see GPU spikes but the temperature was not crazy. the non GPTQ version runs fine. I am using 3090 BTW.

blbx42

Jul 21, 2023

•

edited Jul 21, 2023

Same issue same card. Using ooba webui on commit 63ece46

ooba-webui  | 2023-07-21 18:47:51 ERROR:Failed to load the model.
ooba-webui  | Traceback (most recent call last):
ooba-webui  |   File "/app/server.py", line 68, in load_model_wrapper
ooba-webui  |     shared.model, shared.tokenizer = load_model(shared.model_name, loader)
ooba-webui  |   File "/app/modules/models.py", line 87, in load_model
ooba-webui  |     tokenizer = load_tokenizer(model_name, model)
ooba-webui  |   File "/app/modules/models.py", line 104, in load_tokenizer
ooba-webui  |     tokenizer = AutoTokenizer.from_pretrained(
ooba-webui  |   File "/app/venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 702, in from_pretrained
ooba-webui  |     return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
ooba-webui  |   File "/app/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
ooba-webui  |     return cls._from_pretrained(
ooba-webui  |   File "/app/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2004, in _from_pretrained
ooba-webui  |     tokenizer = cls(*init_inputs, **init_kwargs)
ooba-webui  |   File "/app/venv/lib/python3.10/site-packages/transformers/models/gpt2/tokenization_gpt2.py", line 194, in __init__
ooba-webui  |     with open(merges_file, encoding="utf-8") as merges_handle:
ooba-webui  | TypeError: expected str, bytes or os.PathLike object, not NoneType

TheBloke

Owner Jul 21, 2023

@blob42 the error suggests you don't have file merges.txt. That file does exist in my repo, so please try triggering the download again. Any missing files will be downloaded, without re-downloading anything you do have downloaded successfully.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment