You can deploy this model and start earning money today!

OTAR3088/
CeLLaTe-bioformer-16L-tapt-adapted

Name: CeLLaTe-bioformer-16L-tapt-adapted
Rating: 2.5 (5 reviews)

Views

No views yet

CeLLaTe-bioformer-16L-tapt-adapted

This model is a fine-tuned version of Mardiyyah/bioformer-16l-tokenizer-adapted on an unknown dataset. It achieves the following results on the evaluation set:

More information needed

More information needed

More information needed

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 32
eval_batch_size: 32
seed: 3407
gradient_accumulation_steps: 2
total_train_batch_size: 64
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.06
num_epochs: 50
mixed_precision_training: Native AMP

Training Loss	Epoch	Step	Validation Loss	Accuracy	Perplexity
No log	1.0	15	9.5765	0.0008	14421.3859
No log	2.0	30	7.7367	0.0005	2290.8902
No log	3.0	45	6.8667	0.1422	959.7625
No log	4.0	60	5.8874	0.2321	360.4623
No log	5.0	75	5.2345	0.2851	187.6406
No log	6.0	90	4.7875	0.3268	119.9979
6.7557	7.0	105	4.4625	0.3565	86.6999
6.7557	8.0	120	4.1913	0.3941	66.1101
6.7557	9.0	135	4.0196	0.4181	55.6783
6.7557	10.0	150	3.8186	0.4525	45.5417
6.7557	11.0	165	3.6814	0.4725	39.7034
6.7557	12.0	180	3.5284	0.4950	34.0703
6.7557	13.0	195	3.4597	0.5131	31.8077
3.7563	14.0	210	3.3813	0.5219	29.4080
3.7563	15.0	225	3.2784	0.5396	26.5346
3.7563	16.0	240	3.1944	0.5526	24.3961
3.7563	17.0	255	3.1044	0.5656	22.2966
3.7563	18.0	270	3.1087	0.5623	22.3914
3.7563	19.0	285	2.9939	0.5856	19.9634
3.0567	20.0	300	3.0040	0.5806	20.1659
3.0567	21.0	315	2.9078	0.5942	18.3163
3.0567	22.0	330	2.9290	0.5936	18.7092
3.0567	23.0	345	2.8820	0.6016	17.8503
3.0567	24.0	360	2.8125	0.6057	16.6508
3.0567	25.0	375	2.8499	0.6026	17.2855
3.0567	26.0	390	2.7702	0.6179	15.9623
2.7526	27.0	405	2.7224	0.6209	15.2171
2.7526	28.0	420	2.6848	0.6209	14.6559
2.7526	29.0	435	2.7144	0.6192	15.0951
2.7526	30.0	450	2.6704	0.6236	14.4457
2.7526	31.0	465	2.6776	0.6191	14.5504
2.7526	32.0	480	2.6090	0.6336	13.5857
2.7526	33.0	495	2.6001	0.6337	13.4658
2.5645	34.0	510	2.6302	0.6269	13.8762
2.5645	35.0	525	2.6430	0.6254	14.0549
2.5645	36.0	540	2.6005	0.6343	13.4706