babylm-base10m-roberta

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.4872
• Accuracy: 0.1685

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 200
• training_steps: 20000

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
6.8966	0.1026	200	6.4089	0.0968
6.1095	0.2051	400	6.0478	0.1263
5.9479	0.3077	600	5.9329	0.1364
5.8333	0.4103	800	5.8685	0.1436
5.8139	0.5128	1000	5.8226	0.1442
5.8006	0.6154	1200	5.7855	0.1460
5.7552	0.7179	1400	5.7684	0.1463
5.7202	0.8205	1600	5.7414	0.1503
5.6904	0.9231	1800	5.7274	0.1482
5.6748	1.0256	2000	5.7139	0.1514
5.5622	2.0513	4000	5.6324	0.1577
5.4932	3.0769	6000	5.5928	0.1594
5.4641	4.1026	8000	5.5626	0.1631
5.4467	5.1282	10000	5.5503	0.1648
5.489	6.1538	12000	5.5264	0.1674
5.3918	7.1795	14000	5.5144	0.1677
5.416	8.2051	16000	5.4996	0.1682
5.4	9.2308	18000	5.4923	0.1688

Framework versions

• Transformers 4.50.3
• Pytorch 2.7.1+cu126
• Datasets 3.6.0
• Tokenizers 0.21.4