YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Mostly untested!
RoPE Scaled QLoRA Long Context Extension of Llama-33b (LoRA)
Overview
This is base Llama-33b with minimal additional training to extend the useful context window.
- Context length extended to 16384 by RoPE Scaled Embeddings (Position Interpolation).
- Pretrained for additional 100 steps on 8192 length sequences from the pile dataset.
- The merged model is used as the starting point for training bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA
This is a QLoRA fine-tune
Pretraining took 10 hours on 1x RTX 6000 Ada.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support