Instructions to use SebastianBodza/mpt-30B-qlora-multi_GPU with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SebastianBodza/mpt-30B-qlora-multi_GPU with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SebastianBodza/mpt-30B-qlora-multi_GPU", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("SebastianBodza/mpt-30B-qlora-multi_GPU", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("SebastianBodza/mpt-30B-qlora-multi_GPU", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use SebastianBodza/mpt-30B-qlora-multi_GPU with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SebastianBodza/mpt-30B-qlora-multi_GPU"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SebastianBodza/mpt-30B-qlora-multi_GPU",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/SebastianBodza/mpt-30B-qlora-multi_GPU

SGLang

How to use SebastianBodza/mpt-30B-qlora-multi_GPU with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SebastianBodza/mpt-30B-qlora-multi_GPU" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SebastianBodza/mpt-30B-qlora-multi_GPU",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SebastianBodza/mpt-30B-qlora-multi_GPU" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SebastianBodza/mpt-30B-qlora-multi_GPU",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use SebastianBodza/mpt-30B-qlora-multi_GPU with Docker Model Runner:
```
docker model run hf.co/SebastianBodza/mpt-30B-qlora-multi_GPU
```

mpt-30B-qlora-multi_GPU

Commit History

Update README.md

a748a2b

SebastianBodza commited on Jul 3, 2023

Update README.md

d36cc58

SebastianBodza commited on Jun 30, 2023

Upload 19 files

a7b39ef

SebastianBodza commited on Jun 30, 2023

Create README.md

1283cf1

SebastianBodza commited on Jun 30, 2023

initial commit

481d550

Sebastian Bodza commited on Jun 30, 2023

Commit History

Update README.md a748a2b

Update README.md d36cc58

Upload 19 files a7b39ef

Create README.md 1283cf1

initial commit 481d550

Update README.md

a748a2b

Update README.md

d36cc58

Upload 19 files

a7b39ef

Create README.md

1283cf1

initial commit

481d550