Instructions to use QuantFactory/Mistral-Crab-SFT-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/Mistral-Crab-SFT-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/Mistral-Crab-SFT-GGUF",
	filename="Mistral-Crab-SFT.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/Mistral-Crab-SFT-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/Mistral-Crab-SFT-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/Mistral-Crab-SFT-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/Mistral-Crab-SFT-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Ollama
How to use QuantFactory/Mistral-Crab-SFT-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/Mistral-Crab-SFT-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Mistral-Crab-SFT-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/Mistral-Crab-SFT-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/Mistral-Crab-SFT-GGUF to start chatting

Docker Model Runner
How to use QuantFactory/Mistral-Crab-SFT-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/Mistral-Crab-SFT-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/Mistral-Crab-SFT-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Crab-SFT-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/Mistral-Crab-SFT-GGUF

This is quantized version of THU-KEG/Mistral-Crab-SFT created using llama.cpp

Original Model Card

Model Card for Mistral-Crab-SFT

Large language models (LLMs) struggle to follow instructions with complex constraints in format, length, etc. Following the conventional instruction-tuning practice, previous works conduct post-training on complex instruction-response pairs generated by feeding complex instructions to advanced LLMs. However, even advanced LLMs cannot follow complex instructions well, thus limiting the quality of generated data. In this work, we find that existing datasets inherently contain implicit complex constraints and propose a novel data generation technique, constraint back-translation. Specifically, we take the high-quality instruction-response pairs in existing datasets and only adopt advanced LLMs to add complex constraints already met by the responses to the instructions, which naturally reduces costs and data noise. In the experiments, we adopt Llama3-70B-Instruct to back-translate constraints and create a high-quality complex instruction-response dataset, named CRAB. We present that post-training on CRAB improves multiple backbone LLMs' complex instruction-following ability, evaluated on extensive instruction-following benchmarks. We further find that constraint back-translation also serves as a useful auxiliary training objective in post-training.

📖 Paper: Constraint Back-translation Improves Complex Instruction Following of Large Language Models
🦀 Github: THU/Crab

Model Performance

Models	BaseModel	IFEval	FollowBench(HSR)			AVG
		AVG	L1-L2	L3-L5	AVG
GPT-3.5-turbo	GPT	66.3	74.2	61	66.2	66.3
GPT-4	GPT	81.3	80.4	69.4	73.8	77.6
Vicuna-13b-V1.5	Llama2	50.3	66.3	39.8	50.4	50.4
WizardLM-13B-V1.2	Llama2	51.4	56.5	36.9	44.7	48
Conifer-13B	Llama2	50.2	57.1	40.3	47	48.6
Zephyr-7B-beta	Mistral	45.4	54.8	38.2	44.8	45.1
Conifer-7B	Mistral	53.9	51.9	40.2	44.9	49.4
Conifer-7B-DPO	Mistral	55.7	57	45.4	50	52.9
Llama3 8B	Llama3	31.4	6.8	8.2	7.6	19.5
Llama3-crab	Llama3	46.9	51.2	26.7	36.5	41.7
Llama3-crab + DPO	Llama3	49.7	56.8	38.1	45.5	47.6
Mistral 7B	Mistral	25.2	15.5	6.5	10.1	17.7
Mistral-crab	Mistral	54.5	59.2	32.8	43.3	48.9
Mistral-crab + DPO	Mistral	59.4	59.9	42.5	49.4	54.4

Model Description

Developed by: Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, Juanzi Li
Model type: Text Generation
Language(s) (NLP): English
Finetuned from model [optional]: Mistral-7B-v0.3

Downloads last month: 170

GGUF

Model size

7B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Paper for QuantFactory/Mistral-Crab-SFT-GGUF

Constraint Back-translation Improves Complex Instruction Following of Large Language Models

Paper • 2410.24175 • Published Oct 31, 2024 • 18