metadata
library_name: transformers
tags:
- generated_from_trainer
datasets:
- arrow
model-index:
- name: mixtral_5_toy_model
results: []
mixtral_5_toy_model
This model is a fine-tuned version of on the arrow dataset. It achieves the following results on the evaluation set:
- Loss: 8.6838
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 16
- total_train_batch_size: 256
- total_eval_batch_size: 32
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- training_steps: 780
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0 | 0 | 10.9749 |
| 10.7986 | 1.256 | 50 | 10.5715 |
| 10.3611 | 2.512 | 100 | 10.1266 |
| 9.8898 | 3.768 | 150 | 9.7008 |
| 9.5128 | 5.0 | 200 | 9.4023 |
| 9.169 | 6.256 | 250 | 9.0760 |
| 8.833 | 7.5120 | 300 | 8.8510 |
| 8.5735 | 8.768 | 350 | 8.6894 |
| 8.3489 | 10.0 | 400 | 8.6102 |
| 8.134 | 11.256 | 450 | 8.5842 |
| 7.9193 | 12.512 | 500 | 8.5650 |
| 7.6925 | 13.768 | 550 | 8.5667 |
| 7.4401 | 15.0 | 600 | 8.5921 |
| 7.1474 | 16.256 | 650 | 8.6214 |
| 6.9251 | 17.512 | 700 | 8.6492 |
| 6.7407 | 18.768 | 750 | 8.6718 |
Framework versions
- Transformers 4.53.1
- Pytorch 2.7.0+cu126
- Datasets 3.6.0
- Tokenizers 0.21.1