Model description

A 3 epoch fine-tune of BERT base multilingual uncased on development and humanitarian finance project titles and descriptions from the OECD DAC CRS. Vocabulary of the base model was expanded by 1,059 tokens (1% increase) based on the most prevalent tokens in the CRS that were not present in the base model vocabulary.

Intended uses & limitations

Developed as an experiment to see whether fine-tuning on the CRS would help improve classifier models built on top of this MLM. Although it's built on a multilingual model, an the finetuning texts do include other languages, English will be the most prevalent.

Training and evaluation data

See the OECD DAC CRS project titles and descriptions dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.2133	1.0	58119	1.1296
1.098	2.0	116238	1.0336
1.0441	3.0	174357	0.9958

Framework versions

Transformers 4.38.2
Pytorch 2.0.1
Datasets 2.18.0
Tokenizers 0.15.2

alex-miller/ODABert

ODABert