A-glu-linear-94L

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8067

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 512
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: constant
• training_steps: 1500

Training results

Training Loss	Epoch	Step	Validation Loss
24.2536	0.0270	50	5.9005
22.7293	0.0539	100	5.5937
21.1373	0.0809	150	5.0106
18.1727	0.1078	200	4.4372
16.6833	0.1348	250	3.9876
14.9856	0.1618	300	3.6893
14.0910	0.1887	350	3.4260
12.9818	0.2157	400	3.2223
12.4091	0.2427	450	3.0166
11.4073	0.2696	500	2.8208
10.9365	0.2966	550	2.6682
10.2260	0.3235	600	2.5351
9.8522	0.3505	650	2.4257
9.3503	0.3775	700	2.3227
9.0836	0.4044	750	2.2412
8.7248	0.4314	800	2.1792
8.5722	0.4583	850	2.1275
8.3036	0.4853	900	2.0717
8.1880	0.5123	950	2.0337
8.0157	0.5392	1000	2.0010
7.8838	0.5662	1050	1.9692
7.7710	0.5932	1100	1.9455
7.6902	0.6201	1150	1.9199
7.5579	0.6471	1200	1.8973
7.5209	0.6740	1250	1.8775
7.4264	0.7010	1300	1.8612
7.3811	0.7280	1350	1.8459
7.3058	0.7549	1400	1.8320
7.2745	0.7819	1450	1.8186
7.1992	0.8088	1500	1.8067

Framework versions

• Transformers 5.15.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 5.0.1
• Tokenizers 0.22.2