Erebus-487M-Instruct

Instruction-tuned version of Erebus-487M using LoRA fine-tuning on OpenOrca dataset.

Model Details

• Base Model: erebus-487m-base
• Parameters: 487.8M
• Fine-tuning Method: LoRA (rank=32, alpha=64)
• Training Data: OpenOrca (2.6B tokens, 10,000 steps)
• Final Loss: ~1.95

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("soyrsoyr/erebus-487m-instruct")
tokenizer = AutoTokenizer.from_pretrained("soyrsoyr/erebus-487m-instruct")

prompt = "Q: What is the capital of France?\nA:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Training

• LoRA rank: 32, alpha: 64
• Batch size: 32, gradient accumulation: 8 (effective batch: 256)
• Learning rate: 5e-4
• Device: NVIDIA H200 (71GB MIG slice)
• Training time: ~27 hours

License

Apache 2.0

Editor's Note

This model is poor in just about every way, this was just a proof of concept for the pipeline, better versions will be trained and uploaded soon!