BillSumFineTuned – AI Model by chriskein1 | AlphaNeural AI

You can deploy this model and start earning money today!

chriskein1/
BillSumFineTuned

like

0

text2text-generation

generated_from_trainer

google-t5/t5-small

text-generation-inference

endpoints_compatible

Views

No views yet

Final_Tuning

This model is a fine-tuned version of google-t5/t5-small on our Congressional bill and summarization dataset.

It achieves the following results on the evaluation set:

Loss: 1.4253
Rouge1: 0.2818
Rouge2: 0.2299
Rougel: 0.2729
Rougelsum: 0.2729
Gen Len: 18.9624

Model description

MTSU SoftwareEngineering 2024: Fine-Tuned model for whatsinthebill.ai, a server hosted federal bill summarization model.

Hyperparameters found using Optuna grid search.

Intended uses & limitations

To be used on congressional bills, acts, amendments, etc. as a summarization pipeline.

Training and evaluation data

Trained on the dataset of Congressional bills and summaries my team and I cleaned and collated.

Can be found at https://huggingface.co/datasets/cheaptrix/billsum-US_congress_and_house

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3.28e-05
train_batch_size: 4
eval_batch_size: 4
seed: 42
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 5
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
1.7639	1.0	12429	1.5726	0.2754	0.2199	0.266	0.266	18.9728
1.658	2.0	24858	1.4926	0.2783	0.2246	0.2691	0.269	18.9693
1.6068	3.0	37287	1.4537	0.2819	0.2292	0.2728	0.2728	18.9616
1.5955	4.0	49716	1.4304	0.2812	0.2293	0.2723	0.2723	18.9615
1.5633	5.0	62145	1.4253	0.2818	0.2299	0.2729	0.2729	18.9624

Framework versions

Transformers 4.46.2
Pytorch 2.5.1+cu121
Datasets 3.1.0
Tokenizers 0.20.3