llama_1b_step2_batch_grad_v2

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3255

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 40
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
1.0959	0.0682	50	1.1193
0.8353	0.1363	100	0.9093
0.8185	0.2045	150	0.7860
0.6476	0.2727	200	0.6876
0.6211	0.3408	250	0.6156
0.5487	0.4090	300	0.5546
0.3495	0.4772	350	0.5092
0.4872	0.5453	400	0.4798
0.5069	0.6135	450	0.4531
0.3867	0.6817	500	0.4269
0.3495	0.7498	550	0.4099
0.3642	0.8180	600	0.3949
0.4026	0.8862	650	0.3778
0.3184	0.9543	700	0.3638
0.2109	1.0225	750	0.3620
0.3049	1.0907	800	0.3598
0.2303	1.1588	850	0.3548
0.2837	1.2270	900	0.3498
0.2812	1.2952	950	0.3418
0.212	1.3633	1000	0.3414
0.2756	1.4315	1050	0.3354
0.2508	1.4997	1100	0.3329
0.2125	1.5678	1150	0.3312
0.1492	1.6360	1200	0.3294
0.2738	1.7042	1250	0.3273
0.2589	1.7723	1300	0.3264
0.1845	1.8405	1350	0.3257
0.2307	1.9087	1400	0.3255
0.2417	1.9768	1450	0.3255

Framework versions

• Transformers 4.46.0
• Pytorch 2.1.0+cu118
• Datasets 3.0.2
• Tokenizers 0.20.1