Instructions to use second-state/Mistral-Large-Instruct-2407-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use second-state/Mistral-Large-Instruct-2407-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="second-state/Mistral-Large-Instruct-2407-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("second-state/Mistral-Large-Instruct-2407-GGUF")
model = AutoModelForCausalLM.from_pretrained("second-state/Mistral-Large-Instruct-2407-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use second-state/Mistral-Large-Instruct-2407-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use Docker

docker model run hf.co/second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use second-state/Mistral-Large-Instruct-2407-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "second-state/Mistral-Large-Instruct-2407-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Mistral-Large-Instruct-2407-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

SGLang

How to use second-state/Mistral-Large-Instruct-2407-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "second-state/Mistral-Large-Instruct-2407-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Mistral-Large-Instruct-2407-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "second-state/Mistral-Large-Instruct-2407-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "second-state/Mistral-Large-Instruct-2407-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use second-state/Mistral-Large-Instruct-2407-GGUF with Ollama:
```
ollama run hf.co/second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```

Unsloth Studio

How to use second-state/Mistral-Large-Instruct-2407-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for second-state/Mistral-Large-Instruct-2407-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for second-state/Mistral-Large-Instruct-2407-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for second-state/Mistral-Large-Instruct-2407-GGUF to start chatting

Docker Model Runner
How to use second-state/Mistral-Large-Instruct-2407-GGUF with Docker Model Runner:
```
docker model run hf.co/second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```

Lemonade

How to use second-state/Mistral-Large-Instruct-2407-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull second-state/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Large-Instruct-2407-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

apepkuss79 commited on Aug 2, 2024

Commit

e77c8ff

verified ·

1 Parent(s): 570d2f8

Update models

Browse files

Files changed (15) hide show

.gitattributes +14 -0
Mistral-Large-Instruct-2407-Q8_0-00001-of-00005.gguf +3 -0
Mistral-Large-Instruct-2407-Q8_0-00002-of-00005.gguf +3 -0
Mistral-Large-Instruct-2407-Q8_0-00003-of-00005.gguf +3 -0
Mistral-Large-Instruct-2407-Q8_0-00004-of-00005.gguf +3 -0
Mistral-Large-Instruct-2407-Q8_0-00005-of-00005.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00001-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00002-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00003-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00004-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00005-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00006-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00007-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00008-of-00009.gguf +3 -0
Mistral-Large-Instruct-2407-f16-00009-of-00009.gguf +3 -0

@@ -63,3 +63,17 @@ Mistral-Large-Instruct-2407-Q6_K-00001-of-00004.gguf filter=lfs diff=lfs merge=l
 Mistral-Large-Instruct-2407-Q6_K-00002-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-Q6_K-00003-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-Q6_K-00004-of-00004.gguf filter=lfs diff=lfs merge=lfs -text

 Mistral-Large-Instruct-2407-Q6_K-00002-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-Q6_K-00003-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-Q6_K-00004-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-Q8_0-00001-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-Q8_0-00002-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-Q8_0-00003-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-Q8_0-00004-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-Q8_0-00005-of-00005.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00001-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00002-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00003-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00004-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00005-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00006-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00007-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00008-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral-Large-Instruct-2407-f16-00009-of-00009.gguf filter=lfs diff=lfs merge=lfs -text

+version https://git-lfs.github.com/spec/v1
+oid sha256:62464d29b82c80db1764af8748f900d7a3ebb79d9d631e35a30e7655d83807aa
+size 29843094336

+version https://git-lfs.github.com/spec/v1
+oid sha256:c9c6e6b5b282a6e26f5afec51830362a96c35a5ccddffe25d3d1db068f1bea2d
+size 29788924672

+version https://git-lfs.github.com/spec/v1
+oid sha256:8c7bee017562420d43024699c806aa88db2c86a73b41a2c4811a19f39c6be719
+size 29788875456

+version https://git-lfs.github.com/spec/v1
+oid sha256:1a9e3f7da29626ed6e44e68906aacbebb5eb825059fd4a1ae83254f87d52feb4
+size 29762136960

+version https://git-lfs.github.com/spec/v1
+oid sha256:bc45da493286d31ceb27b0723535cdf960ab31e18f4a631d472d0dfead261e07
+size 11097346048

+version https://git-lfs.github.com/spec/v1
+oid sha256:c1d6f67937b3be8d59679055faf62cba73294356f59b0b28df3489a8fb89da19
+size 29848446880

+version https://git-lfs.github.com/spec/v1
+oid sha256:dbe7bc3661ee5a01e3e79096ae813fe458e8ebebba66a1878c71b9592ec62d26
+size 29797422752

+version https://git-lfs.github.com/spec/v1
+oid sha256:43c9b08f7f42f6d3f68f1ab44a7890d52e38b3fa1e4e1ebad95fa2552b2a4205
+size 29747091296

+version https://git-lfs.github.com/spec/v1
+oid sha256:61da9db695b3b1ca6a19522fb031dba73e13d000b3a9dab28102410ffaed87e0
+size 29822539424

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a002211c1911686e9f5eb238aafd760dc338507e54a2c50ba01e0e76ce5ef3a
+size 29721876192

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ae12aba94e43ccc695204303e6c90cbc5c2a0fdece5dca63aee4513b36fb1b65
+size 29772207968

+version https://git-lfs.github.com/spec/v1
+oid sha256:f5da2a25d122ff3e1d82d2066cf02e2199dbeb01956d72de14590e8cd9294521
+size 29721974624

+version https://git-lfs.github.com/spec/v1
+oid sha256:1fa913877e0140a82b739bbd5503235966294eac83da40b61bbfcfa5767584e1
+size 29746992864

+version https://git-lfs.github.com/spec/v1
+oid sha256:3826f89acac6ec4e6c2378cbb3453bbeb5ecb7307407f542c20ada9e6131e685
+size 7046727040