Instructions to use DiscoResearch/mixtral-7b-8expert with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DiscoResearch/mixtral-7b-8expert with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DiscoResearch/mixtral-7b-8expert", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DiscoResearch/mixtral-7b-8expert", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("DiscoResearch/mixtral-7b-8expert", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DiscoResearch/mixtral-7b-8expert with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DiscoResearch/mixtral-7b-8expert"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/mixtral-7b-8expert",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DiscoResearch/mixtral-7b-8expert

SGLang

How to use DiscoResearch/mixtral-7b-8expert with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DiscoResearch/mixtral-7b-8expert" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/mixtral-7b-8expert",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DiscoResearch/mixtral-7b-8expert" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DiscoResearch/mixtral-7b-8expert",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use DiscoResearch/mixtral-7b-8expert with Docker Model Runner:
```
docker model run hf.co/DiscoResearch/mixtral-7b-8expert
```

Really appreciate the work put into this! I have noticed a change in the model output since first release.

by AARon99 - opened Dec 10, 2023

Discussion

AARon99

Dec 10, 2023

Hello, I saw your discord message in mistral when you first uploaded the converted files. I snagged them right away and did some testing in oobabooga's textgen-webui. I made this post on reddit: https://www.reddit.com/r/Oobabooga/comments/18e5wi7/mixtral7b8expert_working_in_oobabooga_unquantized/
Discussing how interesting the model was.

The next day I wanted to mess around with it more, but noticed the spark I had experienced the previous day was gone. It failed the riddle question, giving answers very similar to other LLMs and the code wasn't formatted right and incorrect. I spent a while trying to pinpoint what happened, and a way to resolve the issue. The .py files you updated resulted in what seems to be significant changes in the models' output. I am now using the original .py files directly and forcing the model to use those local files instead of the .cache files it had overwritten with the updated .py data.

The files and an explanation are here: https://github.com/RandomInternetPreson/MiscFiles/blob/main/DiscoResearch/mixtral-7b-8expert/info.md

Anyway, I know the model is still a bit of an enigma and trying to run it is guess work, but wanted to reach out and let you know, I think you were running the model better with the original .py file.

bjoernp

Disco Research org Dec 10, 2023

•

edited Dec 10, 2023

Yeah it seems to be the case that the change we made (following the equations from some other MoE papers) is not in line with the original implementation. It's interesting that you noticed this in normal use. Benchmarks for the current method were a few % higher. For now, I'll revert these changes in form of a pr and then you can get the new version by calling revision="refs/pr/5".

AARon99

Dec 10, 2023

Oh wow that is extremely interesting!! It seemed that there really was something different about mixtral when I was using it initially, and it took me a while to figure out what happened. It is such a relief to know that the troubleshooting I did was accurate. I was thinking that's why the change was made, because it might have scored higher, I wish I understood the complexity of the issue better. I have personal questions I use with deterministic settings (as deterministic as possible) to see if I want to spend time working with a model and the change to the .py files was like night and day; I had some initial trepidation about making the post because I couldn't point to a score to demonstrate the difference. Thank you again, I'm really interested to see where this model goes and how it changes the open source llm landscape. I'm also really looking forward to trying out your fine-tuned version!!!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment