O0507TESTW4

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -15.8407

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.0441	0.09	10	-3.1083
-5.0046	0.18	20	-6.0412
-6.8874	0.27	30	-8.0540
-9.0261	0.36	40	-9.9891
-10.9861	0.45	50	-12.1696
-12.9302	0.54	60	-13.9751
-13.3926	0.63	70	-13.2579
-13.6313	0.73	80	-13.9466
-14.0192	0.82	90	-13.9525
-13.7502	0.91	100	1.1641
-4.3114	1.0	110	-7.6709
-10.5181	1.09	120	-12.7705
-13.6347	1.18	130	-14.6031
-14.8384	1.27	140	-15.1801
-15.3669	1.36	150	-15.4784
-15.5216	1.45	160	-15.5379
-15.6124	1.54	170	-15.6557
-15.6801	1.63	180	-15.6797
-15.6673	1.72	190	-15.7361
-15.7512	1.81	200	-15.3992
-15.6977	1.9	210	-15.7783
-15.78	1.99	220	-15.7801
-15.7913	2.08	230	-15.7800
-15.8022	2.18	240	-15.8138
-15.8227	2.27	250	-15.8210
-15.8309	2.36	260	-15.8280
-15.8353	2.45	270	-15.8329
-15.8357	2.54	280	-15.8364
-15.8412	2.63	290	-15.8371
-15.8415	2.72	300	-15.8402
-15.8438	2.81	310	-15.8415
-15.846	2.9	320	-15.8407
-15.8458	2.99	330	-15.8407

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0