README.md

---
license: cc-by-nc-4.0
---
This is my first attempt at a qLoRA on Mixtral using the same raw text dataset as the Dendrite model on Mixtral 8x7B Instruct v0.1 with the following notable settings:
```
lora_r 256
lora_alpha 256
..._max_len 256
learning_rate 0.000001
num_train_epochs 2
```

The output is a little less dry but it still maintains the full level of functionality you would expect out of Mixtral instruct. And it still responds to the

```
[INST]
Do a thing
[/INST]
```

format. 
It's not a massive change in the output but I do plan to run a similar but larger data set with more epochs and a higher learning rate.