binomial-babylm-base_seed-42_1e-3
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 3.0685
- Accuracy: 0.4156
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 96
- eval_batch_size: 192
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 768
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 20.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 3.7837 | 1.0 | 608 | 3.6909 | 0.3447 |
| 3.3554 | 2.0 | 1216 | 3.3141 | 0.3835 |
| 3.2025 | 3.0 | 1824 | 3.1991 | 0.3957 |
| 3.1089 | 4.0 | 2432 | 3.1383 | 0.4021 |
| 3.0474 | 5.0 | 3040 | 3.1052 | 0.4062 |
| 2.9929 | 6.0 | 3648 | 3.0808 | 0.4090 |
| 2.9492 | 7.0 | 4256 | 3.0649 | 0.4108 |
| 2.9117 | 8.0 | 4864 | 3.0565 | 0.4123 |
| 2.8725 | 9.0 | 5472 | 3.0508 | 0.4133 |
| 2.8387 | 10.0 | 6080 | 3.0473 | 0.4144 |
| 2.8048 | 11.0 | 6688 | 3.0454 | 0.4151 |
| 2.7698 | 12.0 | 7296 | 3.0486 | 0.4152 |
| 2.7444 | 13.0 | 7904 | 3.0477 | 0.4161 |
| 2.7103 | 14.0 | 8512 | 3.0555 | 0.4154 |
| 2.6783 | 15.0 | 9120 | 3.0599 | 0.4159 |
| 2.645 | 16.0 | 9728 | 3.0685 | 0.4156 |
Framework versions
- Transformers 4.57.3
- Pytorch 2.6.0+cu124
- Datasets 4.4.1
- Tokenizers 0.22.1
- Downloads last month
- 2