Noorhan/mistral-8b-4bit

# Quantized Mistral 8B (4-bit) – `Noorhan/mistral-8b-4bit`

This repository provides a 4-bit quantized version of [`mistralai/Mistral-8B-Instruct-2410`](https://huggingface.co/mistralai/Mistral-8B-Instruct-v0.1) using **AutoGPTQ**, enabling faster inference and reduced memory usage—ideal for RAG-based chatbot applications on CPU/GPU-constrained environments.

---

## 🔧 Quantization Overview

- **Base Model**: `mistralai/Mistral-8B-Instruct-2410`
- **Quantization Method**: 4-bit GPTQ (Group-wise quantization)
- **Tool Used**: [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)
- **Target Device**: Compatible with both CPU and GPU (CUDA)
- **Output Model**: `Noorhan/mistral-8b-4bit`

---

## 💻 Installation

Make sure your environment has the required dependencies:

```bash
pip install auto-gptq
pip install transformers optimum

If using CUDA (GPU support), ensure auto-gptq is compiled with CUDA.

---

🧠 Quantization Process

Here's a minimal example of how the model was quantized using AutoGPTQ:

from transformers import AutoTokenizer, AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "mistralai/Mistral-8B-Instruct-v0.1"

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,        # group-wise quantization
    desc_act=False         # no desc_act
)

# Load the original model in FP16
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Quantize the model
quant_model = AutoGPTQForCausalLM.from_pretrained(
    model_id,
    quantize_config=quantize_config,
    use_triton=False  # set to True if you have Triton installed for faster inference
)

# Save quantized model
quant_model.save_quantized("mistral-8b-4bit")
tokenizer.save_pretrained("mistral-8b-4bit")

---

🚀 Using the Quantized Model

You can now load and use the quantized model as follows:

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

model = AutoGPTQForCausalLM.from_quantized("Noorhan/mistral-8b-4bit", device="cuda:0", use_triton=False)
tokenizer = AutoTokenizer.from_pretrained("Noorhan/mistral-8b-4bit")

prompt = "How does Retrieval-Augmented Generation improve chatbot performance?"

input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

---

🧪 Model Use Cases

This quantized version is ideal for:

• Educational Chatbots
• RAG Pipelines
• Streamlit Deployments
• Resource-constrained environments (e.g., laptops, consumer GPUs)

---

⚠️ Notes and Considerations

• Performance may vary depending on the device and quantization parameters.
• Ensure documents used in RAG are preprocessed and chunked appropriately to maximize retrieval quality.
• Quantization may slightly reduce model accuracy, so validate outputs in your use case.

---

📦 Hugging Face Model

🧠 View the model on Hugging Face

---

👩‍💻 Author

Developed by Noorhan as part of an NLP innovation for educational chatbot deployment.

---

📜 License

This quantized model follows the license of the original model (Mistral AI License).