This model is a fine-tuned version of
sentence-transformers/all-mpnet-base-v2 on the None dataset.
It achieves the following results on the evaluation set:
The model is a binary text classifier based on
sentence-transformers/all-mpnet-base-v2 and fine-tuned on text sourced from national climate policy documents.
The performance of the classifier is very high. On training, the classifier exhibited very good overall performance (F1 ~ 0.95). This performance was evenly balanced between precise identification of true positive classifications (precision ~ 0.95) and a wide net to capture as many true positives as possible (recall ~ 0.95). When tested on real world unseen test data, the performance was still very high (F1 ~ 0.9). However, testing was based on a fairly small out-of-sample dataset. Therefore classification performance will need to further evaluated on deployment.
The combined dataset
GIZ/policy_qa_v0_1 contains ~85k rows. Each row is duplicated twice, to provide varying sequence lengths (denoted by the values 'small', 'medium', and 'large', which correspond to sequence lengths of 60, 85, and 150 respectively - indicated in the 'strategy' column). This effectively means the dataset is reduced by 1/3 in useful size, and the 'strategy' value should be selected based on the use case. For this training, we utilized the 'medium' samples Furthermore, for each row, the 'context' column contains 3 samples of varying quality. The approach used to assess quality and select samples is described below.
The pre-processing operations used to produce the final training dataset were as follows:
