Views
No views yet
sentence-transformers/all-mpnet-base-v2 and then fine-tuned using a 100% few-shot split of the MultiWOZ dataset and a supervised contrastive loss. It is fine-tuned to be used as an in-context example retriever using this few-shot training set, which is provided in the linked repository. More details available in the repo and paper linked within. To cite this model, please consult the citation in the linked GithHub repository README.sentence_transformers and is accurate, though this model is not intended as a general purpose sentence-encoder: it is expecting in-context examples from MultiWOZ to be formatted in a particular way, see the linked repo for details.pip install -U sentence-transformers1from sentence_transformers import SentenceTransformer
2sentences = ["This is an example sentence", "Each sentence is converted"]
3
4model = SentenceTransformer('Brendan/refpydst-100p-referredstates-referred-states')
5embeddings = model.encode(sentences)
6print(embeddings)1from transformers import AutoTokenizer, AutoModel
2import torch
3
4
5#Mean Pooling - Take attention mask into account for correct averaging
6def mean_pooling(model_output, attention_mask):
7 token_embeddings = model_output[0] #First element of model_output contains all token embeddings
8 input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
9 return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
10
11
12# Sentences we want sentence embeddings for
13sentences = ['This is an example sentence', 'Each sentence is converted']
14
15# Load model from HuggingFace Hub
16tokenizer = AutoTokenizer.from_pretrained('Brendan/refpydst-100p-referredstates-referred-states')
17model = AutoModel.from_pretrained('Brendan/refpydst-100p-referredstates-referred-states')
18
19# Tokenize sentences
20encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
21
22# Compute token embeddings
23with torch.no_grad():
24 model_output = model(**encoded_input)
25
26# Perform pooling. In this case, mean pooling.
27sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
28
29print("Sentence embeddings:")
30print(sentence_embeddings)torch.utils.data.dataloader.DataLoader of length 45810 with parameters:{'batch_size': 24, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}sentence_transformers.losses.OnlineContrastiveLoss.OnlineContrastiveLoss{
"epochs": 6,
"evaluation_steps": 15300,
"evaluator": "refpydst.retriever.code.st_evaluator.RetrievalEvaluator",
"max_grad_norm": 1,
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
"optimizer_params": {
"lr": 2e-05
},
"scheduler": "WarmupLinear",
"steps_per_epoch": null,
"warmup_steps": 100,
"weight_decay": 0.01
}SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)