t5-small-finetuned-xsum

This model is a fine-tuned version of t5-small on the scitldr dataset. It achieves the following results on the evaluation set:

• Loss: 3.8686
• Rouge1: 24.7942
• Rouge2: 7.8227
• Rougel: 21.2018
• Rougelsum: 21.2779
• Gen Len: 18.4297

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
No log	1.0	125	4.1327	23.5028	7.9229	19.2335	19.2839	18.5024
No log	2.0	250	4.0197	23.4862	7.3941	19.1734	19.2273	18.4475
No log	3.0	375	3.9659	24.0596	7.6225	20.2649	20.3197	18.2375
4.2188	4.0	500	3.9302	24.323	7.9627	20.7527	20.8616	18.1826
4.2188	5.0	625	3.9060	24.7138	7.9075	21.1786	21.2552	18.1939
4.2188	6.0	750	3.8900	24.696	7.7986	21.161	21.2083	18.2342
4.2188	7.0	875	3.8801	24.8363	7.852	21.2452	21.3039	18.3473
3.991	8.0	1000	3.8736	24.8537	7.9099	21.2259	21.3141	18.3845
3.991	9.0	1125	3.8700	24.7938	7.8088	21.1743	21.2603	18.4233
3.991	10.0	1250	3.8686	24.7942	7.8227	21.2018	21.2779	18.4297

Framework versions

• Transformers 4.27.0.dev0
• Pytorch 2.0.1+cu117
• Datasets 2.12.0
• Tokenizers 0.12.1