MistrAND-7B-v1
MistrAND 7B v1 is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the OASST2 dataset converted to a custom Andalusian Spanish orthography.
This project is part of the Master's Degree Final Thesis titled Conservational AI for endangered languages: a preservation strategy for Tsakonian Greek upon the Andalusian Spanish case, aiming to preserve endangered languages by storing them in QLoRA adapters for unlimited use in the future.
Links
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.9639 | 0.1796 | 1000 | 0.8628 |
| 0.753 | 0.3593 | 2000 | 0.7898 |
| 0.713 | 0.5389 | 3000 | 0.7348 |
| 0.6756 | 0.7185 | 4000 | 0.6888 |
| 0.6665 | 0.8981 | 5000 | 0.6530 |
Framework versions
- PEFT 0.10.1.dev0
- Transformers 4.41.0.dev0
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1
- Downloads last month
- 3
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for jgchaparro/MistrAND-7B-v1
Base model
mistralai/Mistral-7B-v0.1