byt5-finetuned-fixer

This model is a fine-tuned version of google/byt5-small on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1759

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
1.59	1.0	28	0.6260
0.7268	2.0	56	0.3164
0.3774	3.0	84	0.1823
0.2576	4.0	112	0.1705
0.1672	5.0	140	0.1612
0.1413	6.0	168	0.1450
0.1259	7.0	196	0.1288
0.1026	8.0	224	0.1356
0.0735	9.0	252	0.1350
0.0776	10.0	280	0.1388
0.0578	11.0	308	0.1371
0.0468	12.0	336	0.1479
0.0432	13.0	364	0.1562
0.0486	14.0	392	0.1458
0.0572	15.0	420	0.1406
0.0392	16.0	448	0.1467
0.033	17.0	476	0.1580
0.0422	18.0	504	0.1529
0.0287	19.0	532	0.1596
0.0227	20.0	560	0.1628
0.0246	21.0	588	0.1714
0.0249	22.0	616	0.1671
0.0269	23.0	644	0.1688
0.029	24.0	672	0.1718
0.0263	25.0	700	0.1768
0.0261	26.0	728	0.1710
0.0222	27.0	756	0.1722
0.0204	28.0	784	0.1737
0.0216	29.0	812	0.1757
0.0259	30.0	840	0.1759

Framework versions

• Transformers 4.53.0
• Pytorch 2.6.0+cu124
• Datasets 2.14.4
• Tokenizers 0.21.2