|
|
---
|
|
|
language: en
|
|
|
tags:
|
|
|
- jamba
|
|
|
- lora
|
|
|
- chat
|
|
|
- fine-tuning
|
|
|
license: apache-2.0
|
|
|
---
|
|
|
|
|
|
# Jamba Chat LoRA
|
|
|
|
|
|
This is a LoRA fine-tuned version of the Jamba model trained on chat conversations.
|
|
|
|
|
|
## Model Description
|
|
|
|
|
|
- **Base Model:** LaferriereJC/jamba_550M_trained
|
|
|
- **Training Data:** UltraChat dataset
|
|
|
- **Task:** Conversational AI
|
|
|
- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
```python
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
from peft import PeftModel, PeftConfig
|
|
|
|
|
|
# Load the model
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
|
"LaferriereJC/jamba_550M_trained",
|
|
|
trust_remote_code=True
|
|
|
)
|
|
|
model = PeftModel.from_pretrained(model, "your-username/jamba-chat-lora")
|
|
|
|
|
|
# Load the tokenizer
|
|
|
tokenizer = AutoTokenizer.from_pretrained("LaferriereJC/jamba_550M_trained")
|
|
|
|
|
|
# Example usage
|
|
|
text = "User: How are you today?\nAssistant:"
|
|
|
inputs = tokenizer(text, return_tensors="pt")
|
|
|
outputs = model.generate(**inputs, max_length=100)
|
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
|
|
print(response)
|
|
|
```
|
|
|
|
|
|
## Training Details
|
|
|
|
|
|
- **Training Data:** UltraChat dataset (subset)
|
|
|
- **LoRA Config:**
|
|
|
- Rank: 16
|
|
|
- Alpha: 32
|
|
|
- Target Modules: Last layer feed forward experts
|
|
|
- Dropout: 0.1
|
|
|
- **Training Parameters:**
|
|
|
- Learning Rate: 5e-4
|
|
|
- Optimizer: AdamW (32-bit)
|
|
|
- LR Scheduler: Cosine
|
|
|
- Warmup Ratio: 0.03
|
|
|
|
