chriskein1/BillSumFineTuned

Final_Tuning

This model is a fine-tuned version of google-t5/t5-small on our Congressional bill and summarization dataset.

It achieves the following results on the evaluation set:

• Loss: 1.4253
• Rouge1: 0.2818
• Rouge2: 0.2299
• Rougel: 0.2729
• Rougelsum: 0.2729
• Gen Len: 18.9624

Model description

MTSU SoftwareEngineering 2024: Fine-Tuned model for whatsinthebill.ai, a server hosted federal bill summarization model.

Hyperparameters found using Optuna grid search.

Intended uses & limitations

To be used on congressional bills, acts, amendments, etc. as a summarization pipeline.

Training and evaluation data

Trained on the dataset of Congressional bills and summaries my team and I cleaned and collated.

Can be found at https://huggingface.co/datasets/cheaptrix/billsum-US_congress_and_house

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3.28e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
1.7639	1.0	12429	1.5726	0.2754	0.2199	0.266	0.266	18.9728
1.658	2.0	24858	1.4926	0.2783	0.2246	0.2691	0.269	18.9693
1.6068	3.0	37287	1.4537	0.2819	0.2292	0.2728	0.2728	18.9616
1.5955	4.0	49716	1.4304	0.2812	0.2293	0.2723	0.2723	18.9615
1.5633	5.0	62145	1.4253	0.2818	0.2299	0.2729	0.2729	18.9624

Framework versions

• Transformers 4.46.2
• Pytorch 2.5.1+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3