collapse_gemma-2-2b_hs2_replace_iter1_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0637
• Num Input Tokens Seen: 5698680

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3911	0
1.3438	0.0511	5	1.2592	296352
1.1848	0.1021	10	1.1700	589152
1.1275	0.1532	15	1.1329	884504
1.0731	0.2042	20	1.1072	1182424
1.0942	0.2553	25	1.0975	1474984
1.0931	0.3063	30	1.0918	1772592
1.1141	0.3574	35	1.0878	2061504
1.0847	0.4084	40	1.0843	2358064
1.1003	0.4595	45	1.0811	2650896
1.0771	0.5105	50	1.0790	2942864
1.1246	0.5616	55	1.0765	3234512
1.1009	0.6126	60	1.0744	3525376
1.0904	0.6637	65	1.0727	3820376
1.1707	0.7147	70	1.0711	4108240
1.0279	0.7658	75	1.0692	4402208
1.1465	0.8168	80	1.0680	4698016
1.0785	0.8679	85	1.0669	4991408
1.005	0.9190	90	1.0651	5285784
1.0613	0.9700	95	1.0641	5580576

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1