Envoid/Dendrite-8x7Bv1

This is my first attempt at a qLoRA on Mixtral using the same raw text dataset as the Dendrite model on Mixtral 8x7B Instruct v0.1 with the following notable settings:

lora_r 256
lora_alpha 256
..._max_len 256
learning_rate 0.000001
num_train_epochs 2

The output is a little less dry but it still maintains the full level of functionality you would expect out of Mixtral instruct. And it still responds to the

[INST]
Do a thing
[/INST]

format. It's not a massive change in the output but I do plan to run a similar but larger data set with more epochs and a higher learning rate.