Instructions to use MetaIX/GPT4-X-Alpaca-30B-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MetaIX/GPT4-X-Alpaca-30B-4bit")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MetaIX/GPT4-X-Alpaca-30B-4bit")
model = AutoModelForCausalLM.from_pretrained("MetaIX/GPT4-X-Alpaca-30B-4bit")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MetaIX/GPT4-X-Alpaca-30B-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MetaIX/GPT4-X-Alpaca-30B-4bit

SGLang

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MetaIX/GPT4-X-Alpaca-30B-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MetaIX/GPT4-X-Alpaca-30B-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MetaIX/GPT4-X-Alpaca-30B-4bit with Docker Model Runner:
```
docker model run hf.co/MetaIX/GPT4-X-Alpaca-30B-4bit
```

Model does not reply ( Is typing.. ) / MetaIX/GPT4-X-Alpaca-30B-4bit

#14

by ilnurshams - opened May 20, 2023

Discussion

ilnurshams

May 20, 2023

Coul you please advise how can I fix the issue when a model does not reply to my message?
So there is an endless typing ( see pictures ). I've reduced token sizes to min. Chat mode.
Running on Windows 11, RTX4090, 64GB RAM

scepter

May 21, 2023

The same problem here, "'LlamaForCausalLM' object has no attribute 'generate_with_streaming'" in the console. Linux, RTX 4090, 24 VRAM

ilnurshams

May 25, 2023

Okay, I fixed it. I don't know what was the reason.

Fresh install of oobabooga one-click installers
Start start_windows file. Don't dowload any model!
I manually downloaded the necessary model files from https://huggingface.co/MetaIX/GPT4-X-Alpaca-30B-4bit/tree/main
( see picture in attachment ) and put in oobabooga models folder.
I installed pytorch and cuda via Conda ( download and install Conda first, then run Anaconda Prompt ( miniconda3 ) as admin ). There I run these code ( in order to install pytorch and cuda ). You also need to have python 3.10 as I had it already ( download it )!
conda create --name gptq python=3.10 -y
conda activate gptq
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

( I used this tutorial, but it is for linux. https://github.com/qwopqwop200/GPTQ-for-LLaMa
I used only code from tutorial to install pytorch and cuda )

Then I run update_windows file in oobabooga main folder.
Done!

ilnurshams changed discussion status to closed May 25, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment