This model is a fine-tuned version of facebook/w2v-bert-2.0 on the common_voice_16_1 Portuguese subset using 1XRTX 3090. It achieves the following results on the test set:

Wer: 0.10491320595991134
Cer: 0.032070871626631914
Bert Score: 0.9619712047981167
Sentence Similarity: 0.93867844

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 16
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 10
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer	Bert Score
1.2735	1.0	678	0.2292	0.1589	0.0415	0.9498
0.1715	2.0	1356	0.1762	0.1283	0.0344	0.9599
0.1158	3.0	2034	0.1539	0.1100	0.0298	0.9646
0.0821	4.0	2712	0.1362	0.0949	0.0258	0.9703
0.0605	5.0	3390	0.1349	0.0860	0.0236	0.9728
0.0475	6.0	4068	0.1395	0.0871	0.0239	0.9728
0.0355	7.0	4746	0.1487	0.0837	0.0230	0.9739
0.0309	8.0	5424	0.1452	0.0873	0.0240	0.9728
0.0308	9.0	6102	0.1390	0.0843	0.0228	0.9735
0.0239	10.0	6780	0.1282	0.0832	0.0224	0.9739

Evaluation results

Test Wer	Test Cer	Test Bert Score	Runtime	Samples per second
0.09146400542583083	0.02643665913309742	0.9702128323433327	266.8185	35.282

Framework versions

Transformers 4.38.2
Pytorch 2.2.0
Datasets 2.18.0
Tokenizers 0.15.2

tiagomosantos/w2v-bert-2.0-pt_pt_v2

w2v-bert-2.0-pt_pt_v2

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Evaluation results

Framework versions

tiagomosantos/
w2v-bert-2.0-pt_pt_v2