Views
No views yet
We train unsupervised SimCSE on 106 randomly sampled sentences from English Wikipedia, and train supervised SimCSE on the combination of MNLI and SNLI datasets (314k).
Our evaluation code for sentence embeddings is based on a modified version of SentEval. It evaluates sentence embeddings on semantic textual similarity (STS) tasks and downstream transfer tasks. For STS tasks, our evaluation takes the "all" setting, and report Spearman's correlation. See associated paper (Appendix B) for evaluation details.
Uniformity and alignment. We also observe that (1) though pre-trained embeddings have good alignment, their uniformity is poor (i.e., the embeddings are highly anisotropic); (2) post-processing methods like BERT-flow and BERT-whitening greatly improve uniformity but also suffer a degeneration in alignment; (3) unsupervised SimCSE effectively improves uniformity of pre-trained embeddings whereas keeping a good alignment;(4) incorporating supervised data in SimCSE further amends alignment.
1@inproceedings{gao2021simcse,
2 title={{SimCSE}: Simple Contrastive Learning of Sentence Embeddings},
3 author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
4 booktitle={Empirical Methods in Natural Language Processing (EMNLP)},
5 year={2021}
6}
7tianyug@cs.princeton.edu) and Xingcheng (yxc18@mails.tsinghua.edu.cn). If you encounter any problems when using the code, or want to report a bug, you can open an issue. Please try to specify the problem with details so we can help you better and quicker!1from transformers import AutoTokenizer, AutoModel
2
3tokenizer = AutoTokenizer.from_pretrained("princeton-nlp/unsup-simcse-bert-base-uncased")
4
5model = AutoModel.from_pretrained("princeton-nlp/unsup-simcse-bert-base-uncased")
6