Views
No views yet
microsoft/MiniLM-L12-H384-uncased. It was trained on Ms-Marco using loss distillRankNET as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-MiniLM-L12-DistillRankNET")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-MiniLM-L12-DistillRankNET")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 37.40 | 43.98 |
| trec2019 | 96.12 | 74.57 |
| trec2020 | 93.83 | 73.48 |
| fever | 81.21 | 80.95 |
| arguana | 18.48 | 27.97 |
| climate_fever | 27.52 | 20.31 |
| dbpedia | 75.81 | 46.06 |
| fiqa | 43.71 | 36.25 |
| hotpotqa | 85.35 | 66.48 |
| nfcorpus | 57.75 | 34.59 |
| nq | 53.19 | 58.21 |
| quora | 76.34 | 78.62 |
| scidocs | 28.06 | 15.79 |
| scifact | 66.12 | 69.34 |
| touche | 64.33 | 34.46 |
| trec_covid | 87.17 | 70.74 |
| robust04 | 75.25 | 52.28 |
| lotte_writing | 66.66 | 58.11 |
| lotte_recreation | 60.60 | 55.12 |
| lotte_science | 46.01 | 38.34 |
| lotte_technology | 53.36 | 44.41 |
| lotte_lifestyle | 71.62 | 61.69 |
| Mean In Domain | 75.78 | 64.01 |
| BEIR 13 | 58.85 | 49.21 |
| LoTTE (OOD) | 62.25 | 51.66 |