moe_g_medium_5exp
This model is a fine-tuned version of on the arrow dataset. It achieves the following results on the evaluation set:
- Loss: 4.3701
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 48952
- training_steps: 489524
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 7.705 | 0.2043 | 10000 | 7.6665 |
| 6.3026 | 0.4086 | 20000 | 6.2484 |
| 5.4977 | 0.6128 | 30000 | 5.4729 |
| 5.1582 | 0.8171 | 40000 | 5.1324 |
| 4.8639 | 1.0214 | 50000 | 4.9213 |
| 4.7392 | 1.2257 | 60000 | 4.7472 |
| 4.633 | 1.4299 | 70000 | 4.6234 |
| 4.5355 | 1.6342 | 80000 | 4.5328 |
| 4.4732 | 1.8385 | 90000 | 4.4590 |
| 4.2439 | 2.0428 | 100000 | 4.4035 |
| 4.2456 | 2.2471 | 110000 | 4.3664 |
| 4.2315 | 2.4513 | 120000 | 4.3294 |
| 4.2181 | 2.6556 | 130000 | 4.2935 |
| 4.198 | 2.8599 | 140000 | 4.2615 |
| 3.9599 | 3.0642 | 150000 | 4.2492 |
| 3.9936 | 3.2684 | 160000 | 4.2342 |
| 4.0172 | 3.4727 | 170000 | 4.2132 |
| 4.0128 | 3.6770 | 180000 | 4.1920 |
| 4.0039 | 3.8813 | 190000 | 4.1718 |
| 3.7637 | 4.0856 | 200000 | 4.1910 |
| 3.8279 | 4.2898 | 210000 | 4.1811 |
| 3.8062 | 4.4941 | 220000 | 4.1669 |
| 3.8374 | 4.6984 | 230000 | 4.1508 |
| 3.8523 | 4.9027 | 240000 | 4.1341 |
| 3.5732 | 5.1069 | 250000 | 4.1805 |
| 3.6209 | 5.3112 | 260000 | 4.1753 |
| 3.6493 | 5.5155 | 270000 | 4.1628 |
| 3.6775 | 5.7198 | 280000 | 4.1475 |
| 3.697 | 5.9241 | 290000 | 4.1330 |
| 3.3957 | 6.1283 | 300000 | 4.2027 |
| 3.432 | 6.3326 | 310000 | 4.2020 |
| 3.4829 | 6.5369 | 320000 | 4.1904 |
| 3.5107 | 6.7412 | 330000 | 4.1759 |
| 3.5052 | 6.9454 | 340000 | 4.1612 |
| 3.2511 | 7.1497 | 350000 | 4.2497 |
| 3.2859 | 7.3540 | 360000 | 4.2484 |
| 3.3217 | 7.5583 | 370000 | 4.2410 |
| 3.3274 | 7.7626 | 380000 | 4.2309 |
| 3.3318 | 7.9668 | 390000 | 4.2189 |
| 3.0726 | 8.1711 | 400000 | 4.3108 |
| 3.1117 | 8.3754 | 410000 | 4.3149 |
| 3.1165 | 8.5797 | 420000 | 4.3093 |
| 3.1381 | 8.7839 | 430000 | 4.3037 |
| 3.1509 | 8.9882 | 440000 | 4.2963 |
| 2.9392 | 9.1925 | 450000 | 4.3695 |
| 2.9528 | 9.3968 | 460000 | 4.3732 |
| 2.9532 | 9.6011 | 470000 | 4.3727 |
| 2.963 | 9.8053 | 480000 | 4.3708 |
Framework versions
- Transformers 4.51.0
- Pytorch 2.7.0+cu126
- Datasets 3.6.0
- Tokenizers 0.21.1
- Downloads last month
- -