This model is an English-language social orientation tagger is a DistilBERT model trained on the Conversations Gone Awry (CGA) dataset with social orientation labels collected using GPT-4. This model can be used to predict social orientation labels for new conversations. See example usage below or our Github repo for more extensive examples: examples/single_prediction.py or examples/evaluate.py.

See the multilingual version of this model here: tee-oh-double-dee/social-orientation-multilingual

This dataset was created as part of the work described in Social Orientation: A New Feature for Dialogue Analysis, which was accepted to LREC-COLING 2024.

You can make direct use of this social orientation tagger as follows:

Predicted social orientation tags can be prepended to dialog utterances to assist downstream models. For instance, you could convert

to

and then feed these new utterances to a model that predicts if a conversation will succeed or fail. We showed the effectiveness of this strategy in our paper.

There are many settings where it is useful to predict and explain the success or failure of a dialogue. Circumplex theory from psychology models the social orientations (e.g., Warm-Agreeable, Arrogant-Calculating) of conversation participants, which can in turn can be used to predict and explain the outcome of social interactions, such as in online debates over Wikipedia page edits or on the Reddit ChangeMyView forum. This model enables social orientation tagging of dialog utterances.

The prediction set includes: {Assured-Dominant, Gregarious-Extraverted, Warm-Agreeable, Unassuming-Ingenuous, Unassured-Submissive, Aloof-Introverted, Cold, Arrogant-Calculating, Not Available}

See tee-oh-double-dee/social-orientation for details on the training dataset.

We initialize our social orientation tagger weights from the distilbert-base-uncased pre-trained checkpoint from Hugging Face. We use following hyperparameter settings: batch size=32, learning rate=1e-6, we include speaker names before each utterance, we train in 16 bit floating point representation, we use window size of two utterances (i.e., we use the previous utterance's text and the current utterance's text to predict the current utterance's social orientation tag), and we use a weighted loss function to address class imbalance and improve prediction set diversity. The weight \(w'_c\) assigned to each class \(c\) is defined by

$$ w'c = C \cdot \frac{w_c}{\sum{c=1}^C w_c} $$

where \(w_c = \frac{N}{N_c}\), where \(N\) denotes the number of examples in the training set, and \(N_c\) denotes the number of examples in class \(c\) in the training set, and \(C\) is the number of classes. In our case is \(C=9\), including the Not Available class, which is used for all empty utterances.

We evaluate accuracy at the individual utterance level and report the following results:

Without loss weighting, it is possible to achieve an accuracy of 45%.

@misc{morrill2024social,
      title={Social Orientation: A New Feature for Dialogue Analysis}, 
      author={Todd Morrill and Zhaoyuan Deng and Yanda Chen and Amith Ananthram and Colin Wayne Leach and Kathleen McKeown},
      year={2024},
      eprint={2403.04770},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}