Views
No views yet
jhu-clsp/ettin-encoder-17m. It was trained on Ms-Marco using loss distillRankNET as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-17m-DistillRankNET")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-17m-DistillRankNET")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 23.94 | 28.98 |
| trec2019 | 80.25 | 53.28 |
| trec2020 | 86.96 | 53.33 |
| fever | 62.54 | 64.01 |
| arguana | 8.17 | 12.20 |
| climate_fever | 14.74 | 10.74 |
| dbpedia | 56.75 | 30.47 |
| fiqa | 25.77 | 20.02 |
| hotpotqa | 54.49 | 38.89 |
| nfcorpus | 44.88 | 24.70 |
| nq | 30.91 | 35.10 |
| quora | 72.67 | 73.54 |
| scidocs | 14.23 | 7.60 |
| scifact | 44.12 | 46.84 |
| touche | 57.82 | 30.06 |
| trec_covid | 78.54 | 57.16 |
| robust04 | 51.90 | 30.83 |
| lotte_writing | 51.33 | 41.45 |
| lotte_recreation | 43.58 | 38.96 |
| lotte_science | 33.19 | 27.43 |
| lotte_technology | 32.42 | 25.31 |
| lotte_lifestyle | 53.95 | 45.13 |
| Mean In Domain | 63.72 | 45.20 |
| BEIR 13 | 43.51 | 34.72 |
| LoTTE (OOD) | 44.39 | 34.85 |