Mistral-Nemo-Instruct-2407-MLX-2bit

MLX quantized version of Mistral Nemo Instruct 2407.

Quantization

• Method: Q2 (2-bit integer quantization)
• Bits per weight: 2
• Details: Uniform 2-bit integer quantization with group size 64.
• Converted with: mlx-lm

Usage

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("alankessler/Mistral-Nemo-Instruct-2407-MLX-2bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}],
    add_generation_prompt=True,
    tokenize=False,
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Base Model

• Model: Mistral Nemo Instruct 2407
• Parameters: 12B
• Architecture: Mistral Nemo
• License: Apache 2.0