You can deploy this model and start earning money today!

pszemraj/
deberta-v3-base-unified-mcqa-2-choice

Name: deberta-v3-base-unified-mcqa-2-choice
Rating: 2.5 (22 reviews)

generated_from_trainer

pszemraj/unified-mcqa

microsoft/deberta-v3-base

finetune

mit

endpoints_compatible

Views

No views yet

deberta-v3-base-unified-mcqa-2-choice

[!NOTE] If finetuning this further on downstream tasks (classification, etc) you may need to pass ignore_mismatched_sizes=True when loading this model.

This model is a fine-tuned version of microsoft/deberta-v3-base on the pszemraj/unified-mcqa dataset. It achieves the following results on the evaluation set:

Loss: 0.3384
Accuracy: 0.9010
Num Input Tokens Seen: 15919768

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 8
eval_batch_size: 8
seed: 69
gradient_accumulation_steps: 8
total_train_batch_size: 64
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 300
num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Input Tokens Seen
0.3639	0.4579	400	0.3093	0.8700	2444384
0.308	0.9157	800	0.2792	0.8920	4861008
0.2517	1.3732	1200	0.2989	0.8890	7293776
0.2399	1.8310	1600	0.2796	0.8990	9723744
0.1082	2.2885	2000	0.3557	0.9030	12157608
0.1269	2.7463	2400	0.3486	0.8980	14592584

Framework versions

Transformers 4.51.3
Pytorch 2.6.0+cu124
Datasets 3.5.0
Tokenizers 0.21.1

pszemraj/deberta-v3-base-unified-mcqa-2-choice