Views
No views yet
jhu-clsp/ettin-encoder-68m. It was trained on Ms-Marco using loss ADR as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-68m-ADR-MSE")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-68m-ADR-MSE")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 34.13 | 40.57 |
| trec2019 | 95.54 | 74.00 |
| trec2020 | 93.98 | 68.79 |
| fever | 76.77 | 77.02 |
| arguana | 12.56 | 18.62 |
| climate_fever | 16.73 | 12.38 |
| dbpedia | 70.93 | 41.64 |
| fiqa | 42.21 | 34.62 |
| hotpotqa | 82.32 | 63.86 |
| nfcorpus | 53.16 | 32.11 |
| nq | 48.06 | 53.34 |
| quora | 76.63 | 78.38 |
| scidocs | 23.29 | 12.88 |
| scifact | 60.29 | 61.83 |
| touche | 62.63 | 33.60 |
| trec_covid | 89.06 | 74.14 |
| robust04 | 66.57 | 42.06 |
| lotte_writing | 69.14 | 60.07 |
| lotte_recreation | 57.79 | 52.50 |
| lotte_science | 45.91 | 38.86 |
| lotte_technology | 52.18 | 43.12 |
| lotte_lifestyle | 68.75 | 59.76 |
| Mean In Domain | 74.55 | 61.12 |
| BEIR 13 | 54.97 | 45.72 |
| LoTTE (OOD) | 60.06 | 49.40 |