Views
No views yet
jhu-clsp/ettin-encoder-17m. It was trained on Ms-Marco using loss hingeLoss as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-17m-Hinge")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-17m-Hinge")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 28.64 | 34.19 |
| trec2019 | 85.08 | 57.44 |
| trec2020 | 91.83 | 60.57 |
| fever | 65.06 | 66.15 |
| arguana | 12.49 | 18.79 |
| climate_fever | 19.74 | 14.58 |
| dbpedia | 55.32 | 29.80 |
| fiqa | 32.74 | 26.19 |
| hotpotqa | 67.98 | 50.04 |
| nfcorpus | 42.62 | 24.14 |
| nq | 37.44 | 42.07 |
| quora | 74.49 | 75.53 |
| scidocs | 19.25 | 10.30 |
| scifact | 56.89 | 59.72 |
| touche | 61.38 | 32.74 |
| trec_covid | 79.74 | 63.56 |
| robust04 | 56.76 | 33.10 |
| lotte_writing | 52.67 | 43.97 |
| lotte_recreation | 48.91 | 44.02 |
| lotte_science | 36.10 | 30.71 |
| lotte_technology | 40.12 | 32.24 |
| lotte_lifestyle | 59.40 | 50.55 |
| Mean In Domain | 68.52 | 50.73 |
| BEIR 13 | 48.09 | 39.51 |
| LoTTE (OOD) | 48.99 | 39.10 |