Model description

A regression model that embeds text using a fine-tuned bert-base-multilingual-uncased and uses the AutoModelForSequenceClassification class to output predicted percentage climate finance, climate adaptation finance, and climate mitigation finance.

Intended uses & limitations

Intended to regress World Bank project development objectives and abstracts. Not yet validated against project descriptions written outside of the World Bank project API V3.

Training and evaluation data

Data was collected automatically from the World Bank project API V3. For full code on how data was gathered, see: https://github.com/akmiller01/world-bank-climate-regression/blob/main/code/wb_api_climate.py

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-06
train_batch_size: 24
eval_batch_size: 24
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
0.0722	1.0	126	0.0577
0.0481	2.0	252	0.0391
0.04	3.0	378	0.0350
0.0366	4.0	504	0.0332
0.0341	5.0	630	0.0323
0.0323	6.0	756	0.0312
0.0304	7.0	882	0.0312
0.0296	8.0	1008	0.0310
0.0281	9.0	1134	0.0306
0.0268	10.0	1260	0.0310
0.0261	11.0	1386	0.0303
0.0247	12.0	1512	0.0305
0.0243	13.0	1638	0.0310
0.0233	14.0	1764	0.0306
0.0221	15.0	1890	0.0309
0.0222	16.0	2016	0.0307
0.0216	17.0	2142	0.0308
0.0211	18.0	2268	0.0312
0.021	19.0	2394	0.0310
0.0206	20.0	2520	0.0309

Framework versions

Transformers 4.44.2
Pytorch 2.5.0+cu121
Datasets 3.1.0
Tokenizers 0.19.1

alex-miller/wb-climate-regression

wb-climate-regression