This model is a fine-tuned version of openai-community/gpt2 on the gokulsrinivasagan/wiki40b_processed dataset. It achieves the following results on the evaluation set:

Loss: -339.2125
Accuracy: 0.3911

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 24
eval_batch_size: 24
seed: 10
distributed_type: multi-GPU
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 10000
num_epochs: 10
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
-513.3845	0.1132	10000	-505.8524	0.2622
-531.1577	0.2264	20000	-527.8483	0.3266
-540.3256	0.3396	30000	-537.9610	0.3644
-542.4823	0.4528	40000	-540.2148	0.3811
-542.5263	0.5659	50000	-540.4131	0.3911
-542.1284	0.6791	60000	-539.2060	0.3987
-540.9241	0.7923	70000	-537.5175	0.4048
-540.3291	0.9055	80000	-535.8707	0.4095
-539.2526	1.0187	90000	-536.7577	0.4135
-538.7234	1.1319	100000	-536.1422	0.4169
-536.5557	1.2451	110000	-534.4947	0.4200
-535.5725	1.3583	120000	-533.4992	0.4224
-533.4913	1.4715	130000	-530.4818	0.4249
-531.5176	1.5846	140000	-528.1806	0.4272
-530.6571	1.6978	150000	-527.6332	0.4292
-528.2929	1.8110	160000	-526.1181	0.4307
-524.1793	1.9242	170000	-521.5057	0.4325
-523.3159	2.0374	180000	-520.4982	0.4343
-522.8785	2.1506	190000	-519.8093	0.4356
-521.2251	2.2638	200000	-518.1406	0.4367
-516.7446	2.3770	210000	-513.0163	0.4378
-513.1897	2.4902	220000	-511.5113	0.4392
-510.7347	2.6033	230000	-507.7811	0.4402
-508.1415	2.7165	240000	-505.3821	0.4412
-506.7426	2.8297	250000	-503.3896	0.4420
-505.3485	2.9429	260000	-502.4323	0.4431
-504.0267	3.0561	270000	-499.9680	0.4440
-501.2865	3.1693	280000	-498.5437	0.4446
-497.2596	3.2825	290000	-492.2012	0.4452
-492.7128	3.3957	300000	-490.9862	0.4462
-487.9237	3.5089	310000	-482.9363	0.4467
-484.054	3.6220	320000	-481.3902	0.4472
-481.9538	3.7352	330000	-480.8507	0.4478
-479.994	3.8484	340000	-477.3947	0.4483
-477.8339	3.9616	350000	-473.7786	0.4492
-475.5998	4.0748	360000	-474.0926	0.4498
-474.1694	4.1880	370000	-471.2204	0.4503
-476.3406	4.3012	380000	-472.3084	0.4507
-476.1219	4.4144	390000	-475.2962	0.4512
-474.5911	4.5276	400000	-473.3912	0.4517
-476.8605	4.6407	410000	-472.6322	0.4519
-478.9139	4.7539	420000	-477.4765	0.4524
-480.3029	4.8671	430000	-476.6373	0.4529
-480.972	4.9803	440000	-477.7659	0.4535
-482.1678	5.0935	450000	-479.5627	0.4537
-484.7887	5.2067	460000	-483.2962	0.4541
-483.0868	5.3199	470000	-481.5776	0.4545
-479.0181	5.4331	480000	-477.3512	0.4547
-476.8835	5.5462	490000	-475.4365	0.4552
-473.472	5.6594	500000	-471.7900	0.4555
-470.6787	5.7726	510000	-469.4331	0.4557
-470.2033	5.8858	520000	-467.9302	0.4561
-468.79	5.9990	530000	-467.9232	0.4562
-470.2075	6.1122	540000	-468.1803	0.4566
-469.0907	6.2254	550000	-466.8235	0.4568
-467.5284	6.3386	560000	-465.8475	0.4572
-468.3776	6.4518	570000	-467.2909	0.4573
-468.8093	6.5649	580000	-467.1705	0.4577
-468.3296	6.6781	590000	-466.7415	0.4578
-467.5929	6.7913	600000	-466.1070	0.4582
-471.0954	6.9045	610000	-469.1050	0.4583
-470.4095	7.0177	620000	-469.2272	0.4585
-471.4537	7.1309	630000	-468.5949	0.4586
-473.0142	7.2441	640000	-470.2210	0.4590
-473.5846	7.3573	650000	-470.9293	0.4591
-475.7373	7.4705	660000	-474.7934	0.4591
-477.2222	7.5836	670000	-475.6166	0.4595
-477.4298	7.6968	680000	-474.3342	0.4595
-479.5763	7.8100	690000	-477.7514	0.4598
-478.8112	7.9232	700000	-478.4809	0.4599
-482.7851	8.0364	710000	-481.7394	0.4601
-483.1363	8.1496	720000	-481.9806	0.4602
-485.4846	8.2628	730000	-483.7867	0.4604
-488.0089	8.3760	740000	-486.2569	0.4604
-488.2655	8.4892	750000	-487.7609	0.4606
-489.151	8.6023	760000	-487.4376	0.4606
-491.2202	8.7155	770000	-489.5757	0.4610
-491.8805	8.8287	780000	-491.1762	0.4609
-492.839	8.9419	790000	-491.5594	0.4611
-493.9809	9.0551	800000	-493.5830	0.4610
-494.3873	9.1683	810000	-493.9024	0.4613
-494.9278	9.2815	820000	-494.3978	0.4613
-495.2298	9.3947	830000	-495.2346	0.4614
-495.104	9.5079	840000	-495.1164	0.4614
-495.5952	9.6210	850000	-495.5160	0.4615
-495.0771	9.7342	860000	-495.0170	0.4616
-494.9496	9.8474	870000	-495.0371	0.4616
-494.5754	9.9606	880000	-494.6686	0.4616

Framework versions

Transformers 4.42.0.dev0
Pytorch 2.2.2+cu121
Datasets 2.19.0
Tokenizers 0.19.1

gokulsrinivasagan/dcn_clm_wiki40b_scr_hd_4

dcn_clm_wiki40b_scr_hd_4

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

gokulsrinivasagan/
dcn_clm_wiki40b_scr_hd_4