Instructions to use TheBloke/WizardCoder-15B-1.0-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/WizardCoder-15B-1.0-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/WizardCoder-15B-1.0-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/WizardCoder-15B-1.0-GPTQ")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/WizardCoder-15B-1.0-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/WizardCoder-15B-1.0-GPTQ

SGLang

How to use TheBloke/WizardCoder-15B-1.0-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/WizardCoder-15B-1.0-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/WizardCoder-15B-1.0-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardCoder-15B-1.0-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/WizardCoder-15B-1.0-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/WizardCoder-15B-1.0-GPTQ
```

Unable to load using Ooobabooga on CPU, was hoping someone would know why

#10

by split-paradox - opened Jul 8, 2023

Discussion

split-paradox

Jul 8, 2023

I can load the model using cpu vs gpu (Not enough memory on GPU), but when I try to communicate with it, I get the following..

2023-07-08 17:19:25 INFO:Loaded the model in 17.42 seconds.

Traceback (most recent call last):
File "/home/username/oobabooga_linux/text-generation-webui/modules/callbacks.py", line 55, in gentask
ret = self.mfunc(callback=_callback, *args, **self.kwargs)
File "/home/username/oobabooga_linux/text-generation-webui/modules/text_generation.py", line 289, in generate_with_callback
shared.model.generate(**kwargs)
File "/home/username/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 422, in generate
with torch.inference_mode(), torch.amp.autocast(device_type=self.device.type):
File "/home/username/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 411, in device
device = [d for d in self.hf_device_map.values() if d not in {'cpu', 'disk'}][0]
IndexError: list index out of range
Output generated in 0.38 seconds (0.00 tokens/s, 0 tokens, context 67, seed 1158767456)

I have disk and cpu selected. Also have autogptq and gptq_for_llma selected. Still not loading properly. Was wondering if anyone would know why this would be and what settings I'm missing?

modiprabal

Aug 3, 2023

•

edited Aug 3, 2023

@split-paradox Has this issue been fixed? I'm facing the same issue.

split-paradox

Aug 9, 2023

No, it hasn't.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment