Instructions to use maywell/PiVoT-MoE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use maywell/PiVoT-MoE with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="maywell/PiVoT-MoE")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("maywell/PiVoT-MoE")
model = AutoModelForCausalLM.from_pretrained("maywell/PiVoT-MoE")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use maywell/PiVoT-MoE with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "maywell/PiVoT-MoE"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "maywell/PiVoT-MoE",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/maywell/PiVoT-MoE

SGLang

How to use maywell/PiVoT-MoE with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "maywell/PiVoT-MoE" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "maywell/PiVoT-MoE",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "maywell/PiVoT-MoE" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "maywell/PiVoT-MoE",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use maywell/PiVoT-MoE with Docker Model Runner:
```
docker model run hf.co/maywell/PiVoT-MoE
```

Great work on MoE!

by killawhale2 - opened Dec 18, 2023

Discussion

killawhale2

Dec 18, 2023

Your work on MoE is absolutely wonderful!
It appears that you have leveraged the solar 10.7B model, which is fantastic!
If this is true, we should definitely promote your amazing work! :D

maywell

Owner Dec 18, 2023

Hi, Thank you for your interest for my work!
Unfortunately, Solar 10.7B has not been used in this model.
but i am going to use Solar 10.7B in my next work or so.
Since Solar is now on llama architecture it was hard to merge with other mistral based models.

안녕하세요. 작업에 관심 가져주셔서 감사합니다.
안타깝게도, 업스테이지 솔라 모델은 이번 모델에 사용되지 않았습니다 ㅠㅠ
물론 다음 모델에 훌륭한 성능을 지닌 솔라 모델을 사용할 계획은 있습니다.
솔라 모델이 현재 라마 아키텍쳐로만 존재하기에 다른 미스트랄 기반 모델들과 결합하는데에 어려움이 있었습니다.

모델에 관심 가져주셔서 감사합니다 😊

killawhale2

Dec 18, 2023

•

edited Dec 18, 2023

Oh I see, congratulations on your great work!
May I ask what weights were used when initializing the 10.7B experts, as I was unaware of other pretrained/fine-tuned models with 10.7B parameters.

maywell

Owner Dec 18, 2023

Based on my PiVoT-10.7B-Mistral-v0.2-RP, other 10.7B models with RP finetune on huggingface used.

killawhale2 changed discussion status to closed Dec 18, 2023

hunkim

Dec 18, 2023

but i am going to use Solar 10.7B in my next work or so.

@maywell We would love to see the Solar model along with your MOE. Thank you in advance.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment