MuseMistral-v3-35k

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1

Model description

This model generates classical music in ABC notation with only 7B parameters and internet data related to ABC notation.

Training procedure

Trained on free GPUs provided by Kaggle using Axolotl.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 3407
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10
• training_steps: 1000
• mixed_precision_training: Native AMP

Framework versions

• PEFT 0.7.1
• Transformers 4.36.0
• Pytorch 2.1.2
• Datasets 2.16.0
• Tokenizers 0.15.0