metadata
language: en
tags:
- jamba
- lora
- chat
- fine-tuning
license: apache-2.0
Jamba Chat LoRA
This is a LoRA fine-tuned version of the Jamba model trained on chat conversations.
Model Description
- Base Model: LaferriereJC/jamba_550M_trained
- Training Data: UltraChat dataset
- Task: Conversational AI
- Fine-tuning Method: LoRA (Low-Rank Adaptation)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig
# Load the model
model = AutoModelForCausalLM.from_pretrained(
"LaferriereJC/jamba_550M_trained",
trust_remote_code=True
)
model = PeftModel.from_pretrained(model, "your-username/jamba-chat-lora")
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("LaferriereJC/jamba_550M_trained")
# Example usage
text = "User: How are you today?\nAssistant:"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
Training Details
- Training Data: UltraChat dataset (subset)
- LoRA Config:
- Rank: 16
- Alpha: 32
- Target Modules: Last layer feed forward experts
- Dropout: 0.1
- Training Parameters:
- Learning Rate: 5e-4
- Optimizer: AdamW (32-bit)
- LR Scheduler: Cosine
- Warmup Ratio: 0.03