Instructions to use cloudyu/Mixtral_11Bx2_MoE_19B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cloudyu/Mixtral_11Bx2_MoE_19B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="cloudyu/Mixtral_11Bx2_MoE_19B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("cloudyu/Mixtral_11Bx2_MoE_19B")
model = AutoModelForCausalLM.from_pretrained("cloudyu/Mixtral_11Bx2_MoE_19B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cloudyu/Mixtral_11Bx2_MoE_19B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cloudyu/Mixtral_11Bx2_MoE_19B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/Mixtral_11Bx2_MoE_19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cloudyu/Mixtral_11Bx2_MoE_19B

SGLang

How to use cloudyu/Mixtral_11Bx2_MoE_19B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cloudyu/Mixtral_11Bx2_MoE_19B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/Mixtral_11Bx2_MoE_19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cloudyu/Mixtral_11Bx2_MoE_19B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cloudyu/Mixtral_11Bx2_MoE_19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use cloudyu/Mixtral_11Bx2_MoE_19B with Docker Model Runner:
```
docker model run hf.co/cloudyu/Mixtral_11Bx2_MoE_19B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Mixtral MOE 2x10.7B

One of Best MoE Model reviewd by reddit community

MoE of the following models :

kyujinpy/Sakura-SOLAR-Instruct
jeonsworld/CarbonVillain-en-10.7B-v1
Local Test
hf (pretrained=cloudyu/Mixtral_11Bx2_MoE_19B), gen_kwargs: (None), limit: None, num_fewshot: 10, batch_size: auto (32)

Tasks Version Filter n-shot Metric Value Stderr

hellaswag Yaml none 10 acc 0.7142 ± 0.0045

none 10 acc_norm 0.8819 ± 0.0032

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
hellaswag	Yaml	none	10	acc	0.7142	±	0.0045
		none	10	acc_norm	0.8819	±	0.0032

gpu code example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import math

## v2 models
model_path = "cloudyu/Mixtral_11Bx2_MoE_19B"

tokenizer = AutoTokenizer.from_pretrained(model_path, use_default_system_prompt=False)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float32, device_map='auto',local_files_only=False, load_in_4bit=True
)
print(model)
prompt = input("please input prompt:")
while len(prompt) > 0:
  input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

  generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=500,repetition_penalty=1.2
  )
  print(tokenizer.decode(generation_output[0]))
  prompt = input("please input prompt:")

CPU example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import math

## v2 models
model_path = "cloudyu/Mixtral_11Bx2_MoE_19B"

tokenizer = AutoTokenizer.from_pretrained(model_path, use_default_system_prompt=False)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float32, device_map='cpu',local_files_only=False
)
print(model)
prompt = input("please input prompt:")
while len(prompt) > 0:
  input_ids = tokenizer(prompt, return_tensors="pt").input_ids

  generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=500,repetition_penalty=1.2
  )
  print(tokenizer.decode(generation_output[0]))
  prompt = input("please input prompt:")

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	74.41
AI2 Reasoning Challenge (25-Shot)	71.16
HellaSwag (10-Shot)	88.47
MMLU (5-Shot)	66.31
TruthfulQA (0-shot)	72.00
Winogrande (5-shot)	83.27
GSM8k (5-shot)	65.28

Downloads last month: 269

Safetensors

Model size

19B params

Tensor type

F16

Model tree for cloudyu/Mixtral_11Bx2_MoE_19B

Quantizations

6 models

Spaces using cloudyu/Mixtral_11Bx2_MoE_19B 29

Evaluation results

normalized accuracy on AI2 Reasoning Challenge (25-Shot)
test set Open LLM Leaderboard

71.160
normalized accuracy on HellaSwag (10-Shot)
validation set Open LLM Leaderboard

88.470
accuracy on MMLU (5-Shot)
test set Open LLM Leaderboard

66.310
mc2 on TruthfulQA (0-shot)
validation set Open LLM Leaderboard

72.000
accuracy on Winogrande (5-shot)
validation set Open LLM Leaderboard

83.270
accuracy on GSM8k (5-shot)
test set Open LLM Leaderboard

65.280