Instructions to use arcee-ai/Mistral-7B-Instruct-v0.2-expanded with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use arcee-ai/Mistral-7B-Instruct-v0.2-expanded with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="arcee-ai/Mistral-7B-Instruct-v0.2-expanded")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("arcee-ai/Mistral-7B-Instruct-v0.2-expanded")
model = AutoModelForCausalLM.from_pretrained("arcee-ai/Mistral-7B-Instruct-v0.2-expanded", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use arcee-ai/Mistral-7B-Instruct-v0.2-expanded with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "arcee-ai/Mistral-7B-Instruct-v0.2-expanded"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcee-ai/Mistral-7B-Instruct-v0.2-expanded",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/arcee-ai/Mistral-7B-Instruct-v0.2-expanded

SGLang

How to use arcee-ai/Mistral-7B-Instruct-v0.2-expanded with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "arcee-ai/Mistral-7B-Instruct-v0.2-expanded" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcee-ai/Mistral-7B-Instruct-v0.2-expanded",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "arcee-ai/Mistral-7B-Instruct-v0.2-expanded" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "arcee-ai/Mistral-7B-Instruct-v0.2-expanded",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use arcee-ai/Mistral-7B-Instruct-v0.2-expanded with Docker Model Runner:
```
docker model run hf.co/arcee-ai/Mistral-7B-Instruct-v0.2-expanded
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Mistral-7B-Instruct-v0.2-expanded

This method employs mergekit's passthrough method to expand blocks within the "mistralai/Mistral-7B-Instruct-v0.2" model. For every 5th layer, a new layer is added, with the o_proj and down_proj parameters of these added layers initialized to zero, mirroring the approach used in LLaMA Pro.

It's important to note that this configuration has not undergone fine-tuning. So this won't work. Therefore, when fine-tuning, ensure that only every 5th layer is trainable,while all other layers remain frozen.

🧩 Configuration

slices:
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [0, 4]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [3, 4]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [4, 8]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [7, 8]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [8, 12]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [11, 12]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [12, 16]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [15, 16]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [16, 20]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [19, 20]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [20, 24]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [23, 24]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [24, 28]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [27, 28]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0
            
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [28, 32]
  - sources:
      - model: mistralai/Mistral-7B-Instruct-v0.2
        layer_range: [31, 32]
        parameters:
          scale:
            - filter: o_proj
              value: 0.0
            - filter: down_proj
              value: 0.0
            - value: 1.0

merge_method: passthrough
dtype: bfloat16

Function to freeze layers

from transformers import AutoModelForCausalLM

def enable_grad_only_every_nth(model, n):
    """
    This function configures the specified model to enable gradient calculations exclusively for every nth layer, starting 
    from the first layer (0-indexed), to accommodate newly added blocks for training. Concurrently, it freezes the gradients 
    for all other components of the model, including the embedding layers and the model's head. This setup is particularly 
    useful for fine-tuning processes where only a subset of layers are targeted for updates, ensuring efficient training and 
    adaptation of newly integrated layers while maintaining the pre-trained behavior of other model components.
    """

    # Freeze embeddings.
    for param in model.model.embed_tokens.parameters():
        param.requires_grad = False

    # Freeze lm_head.
    for param in model.lm_head.parameters():
        param.requires_grad = False

    # Enable gradients for every nth layer
    layers = model.model.layers  # Access the ModuleList containing the layers

    for index, layer in enumerate(layers):

        if (index + 1) % n == 0:  # Enables gradients for every nth layer, starting from the layer after the 0th
            for param in layer.parameters():
                param.requires_grad = True
        else:
            for param in layer.parameters():
                param.requires_grad = False

model = transformers.AutoModelForCausalLM.from_pretrained(
    "arcee-ai/Mistral-7B-Instruct-v0.2-expanded"
    )
# Update layer gradients, specify the correct value for n based on your model's architecture
n =5
enable_grad_only_every_nth(model, n)

Downloads last month: 93

Safetensors

Model size

9B params

Tensor type

F32

Model tree for arcee-ai/Mistral-7B-Instruct-v0.2-expanded

Quantizations

2 models