Model description

This model measure the coherence of the given text, as defined by similarity to ELI5 texts from Reddit.

Intended uses & limitations

Data filtering and evaluation of pretraining data at scale.

The following hyperparameters were used during training:

learning_rate: 8e-05
train_batch_size: 32
eval_batch_size: 128
seed: 0
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 256
total_eval_batch_size: 1024
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 200
num_epochs: 20

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	0.2774	0.3331	0.5	0.3998	0.6662
0.1863	0.7895	300	0.1846	0.7007	0.6493	0.6580	0.7295
0.1804	1.5789	600	0.1883	0.6808	0.6817	0.6812	0.7157
0.1804	2.3684	900	0.1785	0.7175	0.6490	0.6581	0.7364
0.1781	3.1579	1200	0.1774	0.7201	0.6597	0.6700	0.7410
0.1765	3.9474	1500	0.1795	0.6990	0.6816	0.6878	0.7336
0.174	4.7368	1800	0.1768	0.7214	0.6531	0.6628	0.7393
0.1777	5.5263	2100	0.1838	0.6943	0.6920	0.6931	0.7286
0.1758	6.3158	2400	0.1950	0.7731	0.6021	0.5918	0.7266
0.1749	7.1053	2700	0.1753	0.7147	0.6729	0.6830	0.7423
0.1733	7.8947	3000	0.1748	0.7304	0.6525	0.6621	0.7422
0.1696	8.6842	3300	0.1758	0.7125	0.6767	0.6863	0.7420
0.1723	9.4737	3600	0.1743	0.7243	0.6627	0.6734	0.7437
0.1705	10.2632	3900	0.1740	0.7261	0.6601	0.6706	0.7435
0.1682	11.0526	4200	0.1756	0.7316	0.6481	0.6569	0.7408
0.171	11.8421	4500	0.1734	0.7242	0.6647	0.6756	0.7444
0.1699	12.6316	4800	0.1748	0.7351	0.6473	0.6560	0.7416
0.1696	13.4211	5100	0.1731	0.7235	0.6723	0.6833	0.7464
0.1705	14.2105	5400	0.1738	0.7322	0.6557	0.6659	0.7441
0.1697	15.0	5700	0.1729	0.7205	0.6681	0.6788	0.7438
0.1686	15.7895	6000	0.1726	0.7227	0.6710	0.6819	0.7457
0.1663	16.5789	6300	0.1726	0.7229	0.6707	0.6816	0.7457
0.1684	17.3684	6600	0.1727	0.7213	0.6709	0.6817	0.7450
0.1667	18.1579	6900	0.1726	0.7224	0.6704	0.6813	0.7454
0.1687	18.9474	7200	0.1726	0.7288	0.6656	0.6767	0.7465
0.1675	19.7368	7500	0.1726	0.7268	0.6680	0.6791	0.7465