Instructions to use MetaIX/GPT4-X-Alpasta-30b-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MetaIX/GPT4-X-Alpasta-30b-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MetaIX/GPT4-X-Alpasta-30b-4bit")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MetaIX/GPT4-X-Alpasta-30b-4bit")
model = AutoModelForCausalLM.from_pretrained("MetaIX/GPT4-X-Alpasta-30b-4bit")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MetaIX/GPT4-X-Alpasta-30b-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MetaIX/GPT4-X-Alpasta-30b-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpasta-30b-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MetaIX/GPT4-X-Alpasta-30b-4bit

SGLang

How to use MetaIX/GPT4-X-Alpasta-30b-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MetaIX/GPT4-X-Alpasta-30b-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpasta-30b-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MetaIX/GPT4-X-Alpasta-30b-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MetaIX/GPT4-X-Alpasta-30b-4bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MetaIX/GPT4-X-Alpasta-30b-4bit with Docker Model Runner:
```
docker model run hf.co/MetaIX/GPT4-X-Alpasta-30b-4bit
```

How many tokens can this model handle?

by Tonight223 - opened Apr 30, 2023

Discussion

Tonight223

Apr 30, 2023

I find this model working pretty well with different tasks(except for math and coding) and wonder how many tokens it can handle once?

Yhyu13

Apr 30, 2023

@Amantuer How bad is this model on math or coding? I am just about to run some evaluation CoT on 30B models

Tonight223

Apr 30, 2023

@Yhyu13 you could try to ask what is 15 * 5 and add 5 minus 2, or write a snake game in python, you will know what I mean. Models in this scale are all weak at math and coding. Even GPT-3.5 also make mistakes sometimes, at present only GPT-4 can say is capable of math and coding I think.

asresearch7428

May 1, 2023

•

edited May 1, 2023

I don't know the details of this particular model, but I think it's Llama based, so it probably has a 2048 token limit between prompt and completion. Someone correct me if I'm wrong.

Does depend on the length of fine tuning prompt/completion pairs too, as to how well it should perform at different token counts. Some models have an absolute limit of "x" tokens, but if fine-tuned on shorter pairs, they could perform poorly closer to the limit. My guess, given the fine tuning dataset, it's probably good up to 2048 tokens.

Hope this helps the OP, and whoever else is wondering.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment