Instructions to use michaelfeil/ct2fast-mpt-30b-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use michaelfeil/ct2fast-mpt-30b-chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="michaelfeil/ct2fast-mpt-30b-chat", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("michaelfeil/ct2fast-mpt-30b-chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("michaelfeil/ct2fast-mpt-30b-chat", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use michaelfeil/ct2fast-mpt-30b-chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "michaelfeil/ct2fast-mpt-30b-chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "michaelfeil/ct2fast-mpt-30b-chat",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/michaelfeil/ct2fast-mpt-30b-chat

SGLang

How to use michaelfeil/ct2fast-mpt-30b-chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "michaelfeil/ct2fast-mpt-30b-chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "michaelfeil/ct2fast-mpt-30b-chat",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "michaelfeil/ct2fast-mpt-30b-chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "michaelfeil/ct2fast-mpt-30b-chat",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use michaelfeil/ct2fast-mpt-30b-chat with Docker Model Runner:
```
docker model run hf.co/michaelfeil/ct2fast-mpt-30b-chat
```

OOM with two RTX 3090s w/ NVLink

by spaceman7777 - opened Jul 1, 2023

Discussion

spaceman7777

Jul 1, 2023

Hi,
What sort of hardware have you gotten this running on?

I'm suspicious that, despite my syntax, ctranslate2 is having issues splitting the model between two GPUs.

I'm using:
GeneratorCT2fromHfHub(model_path,
device="cuda",
compute_type="int8_float16",
device_index=[0, 1])

and a pre-downloaded model.

Is it possible to run this on 48GB VRAM? Have you tested splitting across two cards?

Any help would be greatly appreciated, as your model is the best option right now for running mpt-30b on mortal hardware. afaik.

(I've tried calling the ctranslate2 Generator class directly as well. Just looking for a second opinion.)
Thank you :)

michaelfeil

Owner Jul 3, 2023

•

edited Jul 19, 2023

This Libary just wraps downloading of the model from HF, tokenizers, and Ctranslate2 internally.

Seems to me like a feature request to Ctranslate2 then, I am not sure if there is a experimental support for distributed inference, when the model does not fit in single vram.

Update: see issue https://github.com/OpenNMT/CTranslate2/issues/1052

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment