Instructions to use bartowski/Mistral-Large-Instruct-2407-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="bartowski/Mistral-Large-Instruct-2407-GGUF",
	filename="Mistral-Large-Instruct-2407-IQ1_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/Mistral-Large-Instruct-2407-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/Mistral-Large-Instruct-2407-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Ollama
How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Ollama:
```
ollama run hf.co/bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```

Unsloth Studio new

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Mistral-Large-Instruct-2407-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Mistral-Large-Instruct-2407-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for bartowski/Mistral-Large-Instruct-2407-GGUF to start chatting

Pi new

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/Mistral-Large-Instruct-2407-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Large-Instruct-2407-GGUF-Q4_K_M

List all available models

lemonade list

Works good generating python on my 64GB RAM w/ 3090TI 24GB VRAM dev box

by ubergarm - opened Jul 28, 2024

Discussion

ubergarm

Jul 28, 2024

A quick field report for any interested.

Ran it locally and successfully generated a well documented recursive python function for processing YouTube channels to extract video data from all the nested playlists. I fed it some JSON out of the YT APIs and prepended a "system prompt" of You are an experienced software developer with many years experience writing programs and scripts in bash, python, and Linux. Assist the user generating high quality professional well commented code. before my coding request.

./llama-server \
    --model "../models/bartowski/Mistral-Large-Instruct-2407-GGUF/Mistral-Large-Instruct-2407-IQ3_XXS.gguf" \
    --n-gpu-layers 42 \
    --ctx-size 4096 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --threads 24 \
    --flash-attn \
    --mlock \
    --n-predict -1 \
    --host 127.0.0.1 \
    --port 8080

>>> Timings
{
  "predicted_ms": 233351.373,
  "predicted_n": 539,
  "predicted_per_second": 2.3098214211064447,
  "predicted_per_token_ms": 432.9339016697588,
  "prompt_ms": 9810.211,
  "prompt_n": 1775,
  "prompt_per_second": 180.93392690534384,
  "prompt_per_token_ms": 5.526879436619718
}

bartowski

Owner Jul 29, 2024

Awesome!!

You may get faster results if you use a non-IQ quant, since those tend to run slower when not fully offloaded, but I'm glad it's working well for your use case!

ubergarm

Jul 29, 2024

I too have read that non-IQ quants should be faster for CPU heavy inference workloads. I ran a few different models for comparison:

                MODEL                    | size GB | bpw  | Offload Layers | PROMPT EVAL t/s | GENERATION t/s
Mistral-Large-Instruct-2407-IQ3_XXS.gguf |    44   | 3.07 |     42         |     180.93      |     2.31
Mistral-Large-Instruct-2407-Q2_K_L.gguf  |    43   | 2.97 |     42         |     158.66      |     2.53
Mistral-Large-Instruct-2407-Q3_K_M* gguf |    56   | 3.86 |     34         |     135.82      |     1.83

I'm using a 4 month old already burned up Intel i9-14900K CPU (hope to RMA or switch to AMD soon given the current Intel debacle and segfaults despite BIOS/microcode updates), 2x32GB DDR5-5600 RAM, 3090TI FE w/ 24GB VRAM

Cheers and thanks for all the quants!

McUH

Aug 3, 2024

Awesome!!

You may get faster results if you use a non-IQ quant, since those tend to run slower when not fully offloaded, but I'm glad it's working well for your use case!

I suppose it might depend on CPU capability? I have AMD Ryzen 9 7950X3D and use 24 high priority threads for inference (turns out to be noticeably faster than recommended physical cores-1 which would only be 15 active threads) with DDR5 memory and my experience is that when offloading (same model) the file size is the only factor determining the speed. Eg. Q3_K_M is slower than IQ3_M and Q4_K_S is slower than IQ4_XS (despite being non-IQ only little bit bigger than IQ). Memory speed is clearly bottleneck in my case (as the inference speed more or less corresponds to time needed to read the part that is offloaded in RAM).

By the way IQ2_XXS is still pretty good for chatting/role-play despite being only ~ 2bpw and can get >3T/s with 8k context in the above setup. With older 70B models 2bit quants were pretty bad even in this use case but Mistral being larger and better probably makes it retain usability.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment