Cbelem/

scibert-certainty-ordinal

This is a text classification model, fully fine-tuned from a allenai/scibert_scivocab_uncased. It re-uses the main BERT model and fits an ordinal regression head on the [CLS] token. The model is fine-tuned on the certainty labels collected in Wurl et al (2024): Understanding Fine-Grained Distortions in Reports for Scientific Finding. The authors originally collect certainty annotations from humans using a 4-point Likert Scale ranging from (1) Uncertain to (4) Certain. Because the resulting datasets suffer from severe class imbalance, we merge the classes (1) Uncertain and (2) Somewhat Uncertain.

There are 1330 examples in the training set and 334 in the test set. Each example is a sentence long. Examples are filtered from the copenlu/spiced dataset to exhibit final score greater or equal than 4.

The original base rates are as follows:

After combining classes 0 and 1, we obtain the base rates below. Note that this mimicks the procedure adopted in the original paper.

The published model represents one of the 29 models different configurations. The selected model maximizes Quadratic Weighted Kappa (implemented using cohen_kappa with quadratic weights), which is better adapted to ordinal problems, such as ordinal scales. Under this metric, a random model would score 0. We adopt this metric as opposed to accuracy or macro F1 to address class imbalances.

Here is the classification report and test set metrics:

17:44:36  INFO        test  loss=0.9565  acc=0.578  QWK=0.5004
17:44:36  INFO      
              precision    recall  f1-score   support

           0       0.58      0.51      0.54        83
           1       0.47      0.46      0.46       111
           2       0.65      0.71      0.68       140

    accuracy                           0.58       334
   macro avg       0.57      0.56      0.56       334
weighted avg       0.57      0.58      0.57       334

We conduct a hyperparameter sweep of the following hyperp