|
|
--- |
|
|
base_model: |
|
|
- mistralai/Mistral-Large-Instruct-2407 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- mistral |
|
|
- 3bit |
|
|
--- |
|
|
This is a 3bit AutoRound GPTQ version of Mistral-Large-Instruct-2407. |
|
|
This conversion used model-*.safetensors. |
|
|
|
|
|
This quantized model needs at least ~ 50GB + context (~5GB) VRAM. I quantized it so that it could fit 64GB VRAM. |
|
|
|
|
|
Quantization script (it takes around 520 GB RAM and A40 GPU 48GB around 20 hours to convert): |
|
|
``` |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
import torch |
|
|
model_name = "mistralai/Mistral-Large-Instruct-2407" |
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16) |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
|
|
from auto_round import AutoRound |
|
|
|
|
|
bits, group_size, sym = 3, 128, True |
|
|
|
|
|
autoround = AutoRound(model, tokenizer, nsamples=256, iters=512, low_gpu_mem_usage=True, batch_size=4, bits=bits, group_size=group_size, sym=sym, |
|
|
device='cuda') |
|
|
autoround.quantize() |
|
|
output_dir = "./Mistral-Large-Instruct-2407-3bit" |
|
|
autoround.save_quantized(output_dir, format='auto_gptq', inplace=True) |
|
|
|
|
|
``` |
|
|
Evals using lm-eval-harness. |
|
|
|
|
|
``` |
|
|
example command: |
|
|
# !pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git auto-gptq optimum |
|
|
m="VPTQ-community/Mistral-Large-Instruct-2407-v8-k65536-256-woft" |
|
|
!lm_eval --model hf --model_args pretrained={m},dtype=auto --tasks wikitext --num_fewshot 0 --batch_size 1 --output_path ./eval/ |
|
|
``` |
|
|
|
|
|
hf (pretrained=MLDataScientist/Mistral-Large-Instruct-2407-GPTQ-3bit,dtype=auto), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 2 |
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |
|
|
|--------|------:|------|-----:|---------------|---|-----:|---|------| |
|
|
|wikitext| 2|none | 0|bits_per_byte |↓ |0.4103|± | N/A| |
|
|
| | |none | 0|byte_perplexity|↓ |1.3290|± | N/A| |
|
|
| | |none | 0|word_perplexity|↓ |4.5765|± | N/A| |
|
|
|
|
|
vs 3bit VPTQ hf (pretrained=VPTQ-community/Mistral-Large-Instruct-2407-v8-k65536-256-woft,dtype=auto), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 1 |
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |
|
|
|--------|------:|------|-----:|---------------|---|-----:|---|------| |
|
|
|wikitext| 2|none | 0|bits_per_byte |↓ |0.4017|± | N/A| |
|
|
| | |none | 0|byte_perplexity|↓ |1.3211|± | N/A| |
|
|
| | |none | 0|word_perplexity|↓ |4.4324|± | N/A| |
|
|
|
|
|
vs 4bit GPTQ: hf (pretrained=ModelCloud/Mistral-Large-Instruct-2407-gptq-4bit,dtype=auto), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 1: |
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |
|
|
|--------|------:|------|-----:|---------------|---|-----:|---|------| |
|
|
|wikitext| 2|none | 0|bits_per_byte |↓ |0.3536|± | N/A| |
|
|
| | |none | 0|byte_perplexity|↓ |1.2777|± | N/A| |
|
|
| | |none | 0|word_perplexity|↓ |3.7082|± | N/A| |
|
|
|
|
|
vs 4bit VPTQ |
|
|
hf (pretrained=VPTQ-community/Mistral-Large-Instruct-2407-v8-k65536-65536-woft,dtype=auto), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 1 |
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |
|
|
|--------|------:|------|-----:|---------------|---|-----:|---|------| |
|
|
|wikitext| 2|none | 0|bits_per_byte |↓ |0.3415|± | N/A| |
|
|
| | |none | 0|byte_perplexity|↓ |1.2671|± | N/A| |
|
|
| | |none | 0|word_perplexity|↓ |3.5463|± | N/A| |
|
|
|
|
|
vs exl2 4bpw (I think the tests are different) |
|
|
| |Wikitext| C4 |FineWeb|Max VRAM| |
|
|
|-------------|--------|-----|-------|--------| |
|
|
|EXL2 4.00 bpw| 2.885 |6.484| 6.246 |60.07 GB| |
|
|
|
