|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- quantization |
|
|
- bitsandbytes |
|
|
- 4-bit |
|
|
- nf4 |
|
|
- double-quant |
|
|
- mcqa |
|
|
--- |
|
|
# Model Card for `Kikinoking/MNLP_M3_quantized_model` |
|
|
|
|
|
A 4-bit double-quantized (NF4 + nested quant) version of the MNLP_M3_mcqa_model, compressed with bitsandbytes. This model answers multiple-choice questions (MCQA) with minimal GPU memory usage. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model ID:** `Kikinoking/MNLP_M3_quantized_model` |
|
|
- **Quantization:** 4-bit NF4 + nested quantization (`bnb_4bit_use_double_quant=True`) |
|
|
- **Base model:** `aidasvenc/MNLP_M3_mcqa_model` |
|
|
- **Library:** Transformers + bitsandbytes |
|
|
- **Task:** Multiple-choice question answering (MCQA) |
|
|
|
|
|
## Usage |
|
|
|
|
|
Load and run inference in just a few lines: |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
import torch |
|
|
|
|
|
model_id = "Kikinoking/MNLP_M3_quantized_model" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_id, |
|
|
device_map="auto", |
|
|
torch_dtype="auto", |
|
|
low_cpu_mem_usage=True |
|
|
).eval() |
|
|
|
|
|
prompt = "What is the capital of France ?\nA) Lyon B) Marseille C) Paris D) Toulouse\nAnswer: " |
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
|
|
|
with torch.inference_mode(): |
|
|
output = model.generate(**inputs, max_new_tokens=1) |
|
|
|
|
|
print("Answer:", tokenizer.decode(output[0], skip_special_tokens=True)) |
|
|
|
|
|
##How It Was Built |
|
|
|
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig |
|
|
import torch |
|
|
|
|
|
base_id = "aidasvenc/MNLP_M3_mcqa_model" |
|
|
qcfg = BitsAndBytesConfig( |
|
|
load_in_4bit=True, |
|
|
bnb_4bit_quant_type="nf4", |
|
|
bnb_4bit_compute_dtype=torch.bfloat16, |
|
|
bnb_4bit_use_double_quant=True |
|
|
) |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(base_id) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
base_id, |
|
|
quantization_config=qcfg, |
|
|
device_map="auto", |
|
|
torch_dtype="auto" |
|
|
) |
|
|
|
|
|
# Push to Hugging Face Hub |
|
|
model.push_to_hub("Kikinoking/MNLP_M3_quantized_model", private=True) |
|
|
tokenizer.push_to_hub("Kikinoking/MNLP_M3_quantized_model") |
|
|
|
|
|
print("VRAM used (MiB):", torch.cuda.memory_reserved()/1024**2) |
|
|
|
