sentence-transformers/LaBSE pre-trained on an instructional question-and-answer dataset. Evaluated on
Precision at K metrics and
Mean reciprocal rank.
Precision at K is a simple metric to understand and implement, but it has an important disadvantage - it does not take into account the order of elements in the "top". So, if we guessed only one item out of ten, it doesn't matter whether it was on the first or the last place - inline_formula in any case. It is obvious that the first variant is much better.
ean reciprocal rank equal to the reverse rank of the first correctly guessed item. Mean reciprocal rank varies in the range [0,1] and takes into account the position of items. Unfortunately, it does this only for one item - the 1st correctly predicted item, ignoring all subsequent items.
1p@1: 52 %
2p@3: 66 %
3p@5: 73 %
4p@10: 79 %
5p@15: 82 %
6MRR: 62 %
1import torch
2from transformers import AutoTokenizer, AutoModel
3tokenizer = AutoTokenizer.from_pretrained("zjkarina/LaBSE-instructDialogs")
4model = AutoModel.from_pretrained("zjkarina/LaBSE-instructDialogs")
5sentences = ["List 5 reasons why someone should learn to code", "Describe the sound of the wind on a sunny day."]
6encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=64, return_tensors='pt')
7with torch.no_grad():
8 model_output = model(**encoded_input)
9embeddings = model_output.pooler_output
10embeddings = torch.nn.functional.normalize(embeddings)
11print(embeddings)
