Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 100
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	1	2.9106
0.9734	2.0	2	2.5432
0.9734	3.0	3	2.5525
0.9213	4.0	4	2.3557
0.9213	5.0	5	2.4531
1.0261	6.0	6	2.2645
1.0261	7.0	7	2.3934
1.3619	8.0	8	2.0998
1.3619	9.0	9	1.9944
0.8367	10.0	10	1.9301
0.8367	11.0	11	1.8057
0.6422	12.0	12	1.8467
0.6422	13.0	13	1.8696
0.6748	14.0	14	1.9901
0.6748	15.0	15	1.9514
0.4253	16.0	16	1.9247
0.4253	17.0	17	1.9163
0.4108	18.0	18	1.9326
0.4108	19.0	19	1.9893
0.4195	20.0	20	2.0361
0.4195	21.0	21	2.0074
0.4775	22.0	22	2.0498
0.4775	23.0	23	1.8154
0.3208	24.0	24	1.9197
0.3208	25.0	25	1.9540
0.5455	26.0	26	1.8042
0.5455	27.0	27	1.8672
0.4088	28.0	28	1.7870
0.4088	29.0	29	1.7510
0.3953	30.0	30	1.7780
0.3953	31.0	31	1.7736
0.5282	32.0	32	1.9123
0.5282	33.0	33	2.1976
0.511	34.0	34	2.2451
0.511	35.0	35	2.3681
0.5864	36.0	36	2.3280
0.5864	37.0	37	2.3096
0.4865	38.0	38	2.3100
0.4865	39.0	39	2.2886
0.4169	40.0	40	2.2931
0.4169	41.0	41	2.2026
0.3892	42.0	42	2.2682
0.3892	43.0	43	1.9952
0.2546	44.0	44	2.0938
0.2546	45.0	45	2.1655
0.3692	46.0	46	2.1784
0.3692	47.0	47	2.3851
0.2428	48.0	48	2.3919
0.2428	49.0	49	2.4402
0.3214	50.0	50	2.4694
0.3214	51.0	51	2.4446
0.1496	52.0	52	2.3300
0.1496	53.0	53	2.1650
0.1285	54.0	54	2.0729
0.1285	55.0	55	2.1043
0.2943	56.0	56	2.0690
0.2943	57.0	57	2.1732
0.5513	58.0	58	2.2477
0.5513	59.0	59	2.2747
0.2935	60.0	60	2.5255
0.2935	61.0	61	2.6288
0.2646	62.0	62	2.8252
0.2646	63.0	63	2.8183
0.2454	64.0	64	2.8435
0.2454	65.0	65	2.7582
0.2294	66.0	66	2.7930
0.2294	67.0	67	2.7311
0.1822	68.0	68	2.6562
0.1822	69.0	69	2.5528
0.2676	70.0	70	2.4556
0.2676	71.0	71	2.5049
0.2179	72.0	72	2.4991
0.2179	73.0	73	2.3995
0.3808	74.0	74	2.4511
0.3808	75.0	75	2.3586
0.2054	76.0	76	2.4260
0.2054	77.0	77	2.3135
0.2335	78.0	78	2.3640
0.2335	79.0	79	2.3055
0.1502	80.0	80	2.2088
0.1502	81.0	81	2.3361
0.2101	82.0	82	2.4369
0.2101	83.0	83	2.4233
0.3398	84.0	84	2.4692
0.3398	85.0	85	2.4868
0.1529	86.0	86	2.4996
0.1529	87.0	87	2.6840
0.1963	88.0	88	2.5101
0.1963	89.0	89	2.4586
0.1637	90.0	90	2.5021
0.1637	91.0	91	2.4644
0.2337	92.0	92	2.5042
0.2337	93.0	93	2.3934
0.088	94.0	94	2.4024
0.088	95.0	95	2.3115
0.0937	96.0	96	2.4827
0.0937	97.0	97	2.3294
0.2625	98.0	98	2.3843
0.2625	99.0	99	2.3754
0.2198	100.0	100	2.3297

Framework versions

Transformers 4.41.1
Pytorch 2.3.0+cu121
Datasets 2.19.1
Tokenizers 0.19.1