---
library_name: transformers
base_model: openlm-research/open_llama_3b_v2
tags:
  - 4bit
  - bnb
  - nf4
  - qlora
  - llama
quantized_by: ping98k
---

# Open LLaMA 3B v2 — 4-bit NF4

4-bit quantized version of [openlm-research/open_llama_3b_v2](https://huggingface.co/openlm-research/open_llama_3b_v2) using bitsandbytes NF4, ready for QLoRA fine-tuning.

## Quantization Details

| Parameter | Value |
|---|---|
| Quant method | bitsandbytes NF4 |
| Double quant | Yes |
| Compute dtype | bfloat16 |
| Model size | ~1.93 GB |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ping98k/open_llama_3b_v2_4bit", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("ping98k/open_llama_3b_v2_4bit")
```

### QLoRA Fine-tuning

```python
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
```