---
license: apache-2.0
base_model: Llama-2-7B-bf16-sharded
model-index:
- name: llama_ft
  results: []
---


# llama_ft

This model is a fine-tuned version of [Llama-2-7B-bf16-sharded](https://huggingface.co/TinyPixel/Llama-2-7B-bf16-sharded) on a grocery cart dataset.

## Intended uses & limitations

The model helps to tell to what type of grocery does the following items belong to.


## Training procedure
Fine tuning techniques like Qlora and PEFT have been used to train the model on the dataset on a single gpu , and the adapters are then finally merged with the model.

    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16

The loading configurations of the model

### Training hyperparameters

The following are the LORA configs-->

    lora_alpha = 16
    lora_dropout = 0.1
    lora_r = 64

    peft_config = LoraConfig(
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    r=lora_r,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","v_proj"]
    )

The following are the training configs -->

    per_device_train_batch_size = 4
    gradient_accumulation_steps = 4
    optim = "paged_adamw_32bit"
    save_steps = 10
    logging_steps = 1
    learning_rate = 2e-4
    max_grad_norm = 0.3
    max_steps = 120
    warmup_ratio = 0.03
    lr_scheduler_type = "constant"