---
language: en
tags:
- jamba
- lora
- chat
- fine-tuning
license: apache-2.0
---

# Jamba Chat LoRA

This is a LoRA fine-tuned version of the Jamba model trained on chat conversations.

## Model Description

- **Base Model:** LaferriereJC/jamba_550M_trained
- **Training Data:** UltraChat dataset
- **Task:** Conversational AI
- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

# Load the model
model = AutoModelForCausalLM.from_pretrained(
    "LaferriereJC/jamba_550M_trained",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(model, "your-username/jamba-chat-lora")

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("LaferriereJC/jamba_550M_trained")

# Example usage
text = "User: How are you today?\nAssistant:"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

## Training Details

- **Training Data:** UltraChat dataset (subset)
- **LoRA Config:**
  - Rank: 16
  - Alpha: 32
  - Target Modules: Last layer feed forward experts
  - Dropout: 0.1
- **Training Parameters:**
  - Learning Rate: 5e-4
  - Optimizer: AdamW (32-bit)
  - LR Scheduler: Cosine
  - Warmup Ratio: 0.03