Contrastive user encoder (single post)
This model is a DistilBertModel trained by fine-tuning distilbert-base-uncased on author-based triplet loss.
Details
Training and evaluation details are provided in our EMNLP Findings paper:
- Rocca, R., & Yarkoni, T. (2022), Language as a fingerprint: Self-supervised learning of user encodings using transformers, to appear in Findings of the Association for Computational Linguistics: EMNLP 2022
Training
We fine-tuned DistilBERT on triplets consisting of:
- a single Reddit submission from a given user (the "anchor") - see
rbroc/contrastive-user-encoder-multipost for a model trained on aggregated embeddings of multiple anchors;
- an additional post from the same user (a "positive example");
- a post from a different, randomly selected user (the "negative example")
To compute the loss, we use [CLS] encoding of the anchor, positive example and negative example from the last layer of the DistilBERT encoder. We optimize for \(max(||f(a) - f(n)|| - ||f(a) - f(p)|| + \alpha,0)\)
where:
- \( f(a)\) is the [CLS] encoding of the anchor;
- \( f(n) \) is the [CLS] encoding of the negative example;
- \( f(p) \) is the [CLS] encoding of the positive example;
- \( \alpha \) is a tunable parameter called margin. Here, we tuned this to \( \alpha = 1.0\)
Evaluation and usage
The model yields performance advantages downstream user-based classification tasks.
We encourage usage and benchmarking on tasks involving:
- prediction of user traits (e.g., personality);
- extraction of user-aware text encodings (e.g., style modeling);
- contextualized text modeling, where standard text representations are complemented with compact user representations
Limitations
Being exclusively trained on Reddit data, our models probably overfit to linguistic markers and traits which are relevant to characterizing the Reddit user population, but less salient in the general population. Domain-specific fine-tuning may be required before deployment.
Furthermore, our self-supervised approach enforces little or no control over biases, which models may actively use as part of their heuristics in contrastive and downstream tasks.
