Instructions to use MetaIX/GPT4-X-Alpaca-30B-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MetaIX/GPT4-X-Alpaca-30B-4bit")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MetaIX/GPT4-X-Alpaca-30B-4bit")
model = AutoModelForCausalLM.from_pretrained("MetaIX/GPT4-X-Alpaca-30B-4bit")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MetaIX/GPT4-X-Alpaca-30B-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MetaIX/GPT4-X-Alpaca-30B-4bit

SGLang

How to use MetaIX/GPT4-X-Alpaca-30B-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MetaIX/GPT4-X-Alpaca-30B-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MetaIX/GPT4-X-Alpaca-30B-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpaca-30B-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MetaIX/GPT4-X-Alpaca-30B-4bit with Docker Model Runner:
```
docker model run hf.co/MetaIX/GPT4-X-Alpaca-30B-4bit
```

sha256sum not matching for one file gpt4-x-alpaca-30b-128g-4bit.safetensors

by spaceman7777 - opened Apr 23, 2023

Discussion

spaceman7777

Apr 23, 2023

So, the hash sums are fine for the rest of the files, but for gpt4-x-alpaca-30b-128g-4bit.safetensors list:
listed in the repo web page:
f770adef16fe3e7ee53c8fbb53c461bf073b3cfb9fea70c7b90dbb4205480d5d
git lfs:
303d325f8b442c8229fc0695c9900f37caa366751ac44d6c2fd8114fd249c0e8
direct download:
49c4138a87e2615cae11a621247763808e540a8053c1be3162f60212811435c7

I've tried it multiple times, and I continue to get issues, but only with that specific file.
It throws an error in git when fetching as well.
$ git lfs pull
expected OID f770adef16fe3e7ee53c8fbb53c461bf073b3cfb9fea70c7b90dbb4205480d5d, got 49c4138a87e2615cae11a621247763808e540a8053c1be3162f60212811435c7 after 18050060784 bytes written

Anyone else see this?

eldr

Apr 30, 2023

This comment has been hidden

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment