collapse_gemma-2-2b_hs2_replace_iter1_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0637
• Num Input Tokens Seen: 5709936

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3911	0
1.2753	0.0511	5	1.2592	285512
1.2151	0.1021	10	1.1718	578296
1.1556	0.1532	15	1.1339	873440
1.1448	0.2042	20	1.1079	1168560
1.0673	0.2553	25	1.0980	1463952
1.1503	0.3063	30	1.0929	1754024
1.0342	0.3574	35	1.0886	2046160
1.0634	0.4084	40	1.0852	2341224
1.1423	0.4595	45	1.0825	2635056
1.0152	0.5105	50	1.0796	2927424
1.0929	0.5616	55	1.0768	3221968
1.1003	0.6126	60	1.0747	3519568
1.0713	0.6637	65	1.0726	3816688
1.0621	0.7147	70	1.0710	4117768
1.0789	0.7658	75	1.0696	4418488
1.1539	0.8168	80	1.0683	4709408
1.1031	0.8679	85	1.0670	5000912
1.0455	0.9190	90	1.0654	5295112
1.0684	0.9700	95	1.0642	5591032

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1