The model maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. The easiest way is to simply measure the cosine distance between two sentences. Sentences that are close to each other in meaning, will have a small cosine distance and a similarity close to 1. The model is trained in such a way that similar sentences in different languages should also be close to each other. Ideally, an English-Norwegian sentence pair should have high similarity.
This release is a non-generative encoder model whose outputs are vectors/scores rather than language or media. Its intended functionality is limited to representation, retrieval, ranking, or classification support. On that basis, the release is preliminarily assessed as not falling within the provider obligations for GPAI models under the EU AI Act definitions, subject to legal confirmation if capability scope or marketed generality changes. For more information, see the Model Documentation Form here.
1from sentence_transformers import SentenceTransformer
23# Download from the 🤗 Hub4model = SentenceTransformer("NbAiLab/nb-sbert-v2-base")5# Run inference6sentences =[7"This is a Norwegian boy",8"Dette er en norsk gutt"9]1011embeddings = model.encode(sentences)12print(embeddings.shape)13# (2, 768)1415# Get the similarity scores for the embeddings16similarities = model.similarity(embeddings, embeddings)17print(similarities)18# tensor([[1.0000, 0.8287],19# [0.8287, 1.0000]])
Direct Usage (Transformers)
Without sentence-transformers, you can still use the model. First, you pass in your input through the transformer model, then you have to apply the right pooling-operation on top of the contextualized word embeddings.
Click to see the direct usage in Transformers
python
1import torch
23from sklearn.metrics.pairwise import cosine_similarity
4from transformers import AutoTokenizer, AutoModel
56#Mean Pooling - Take attention mask into account for correct averaging7defmean_pooling(model_output, attention_mask):8 token_embeddings = model_output[0]#First element of model_output contains all token embeddings9 input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()10return torch.sum(token_embeddings * input_mask_expanded,1)/ torch.clamp(input_mask_expanded.sum(1),min=1e-9)111213# Sentences we want sentence embeddings for14sentences =["This is a Norwegian boy","Dette er en norsk gutt"]1516# Load model from HuggingFace Hub17tokenizer = AutoTokenizer.from_pretrained('NbAiLab/nb-sbert-v2-base')18model = AutoModel.from_pretrained('NbAiLab/nb-sbert-v2-base')1920# Tokenize sentences21encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')2223# Compute token embeddings24with torch.no_grad():25 model_output = model(**encoded_input)2627# Perform pooling. In this case, mean pooling.28embeddings = mean_pooling(model_output, encoded_input['attention_mask'])29print(embeddings.shape)30# torch.Size([2, 768])3132similarity = cosine_similarity(embeddings[0].reshape(1,-1), embeddings[1].reshape(1,-1))33print(similarity)34# This should give 0.8287 in the example above.
Approximate statistics based on the first 1000 samples:
anchor
positive
negative
type
string
string
string
details
min: 5 tokens
mean: 20.91 tokens
max: 130 tokens
min: 5 tokens
mean: 20.91 tokens
max: 130 tokens
min: 5 tokens
mean: 14.14 tokens
max: 39 tokens
Samples:
anchor
positive
negative
Det som følger er mindre en glid nedover en glatt skråning enn et profesjonelt skred som resulterer i enten en oppsigelse eller en smal flukt til neste drømmejobb, der, selvfølgelig, syklusen gjentas igjen.
Syklusen gjentar seg ved neste jobb.
Syklusen gjentar seg sjelden ved neste jobb.
Syklusen gjentar seg ved neste jobb.
Det som følger er mindre en glid nedover en glatt skråning enn et profesjonelt skred som resulterer i enten en oppsigelse eller en smal flukt til neste drømmejobb, der, selvfølgelig, syklusen gjentas igjen.
Syklusen gjentar seg sjelden ved neste jobb.
The public areas are spectacular, the rooms a bit less so, but a long-awaited renovation was carried out in 1998.
The rooms are nice, but the public area is in a league of it's own.
The public area was fine, but the rooms were really something else.
1@inproceedings{reimers-2019-sentence-bert,
2 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
3 author = "Reimers, Nils and Gurevych, Iryna",
4 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
5 month = "11",
6 year = "2019",
7 publisher = "Association for Computational Linguistics",
8 url = "https://arxiv.org/abs/1908.10084",
9}
MultipleNegativesRankingLoss
bibtex
1@misc{henderson2017efficient,
2 title={Efficient Natural Language Response Suggestion for Smart Reply},
3 author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
4 year={2017},
5 eprint={1705.00652},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
NbAiLab/nb-bert-base
bibtex
1@inproceedings{kummervold-etal-2021-operationalizing,
2 title = {Operationalizing a National Digital Library: The Case for a {N}orwegian Transformer Model},
3 author = {Kummervold, Per E and
4 De la Rosa, Javier and
5 Wetjen, Freddy and
6 Brygfjeld, Svein Arne},
7 booktitle = {Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)},
8 year = {2021},
9 address = {Reykjavik, Iceland (Online)},
10 publisher = {Linköping University Electronic Press, Sweden},
11 url = {https://huggingface.co/papers/2104.09617},
12 pages = {20--29},
13 abstract = {In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmål and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.},
14}
Citing & Authors
The model was trained by Victoria Handford and Lucas Georges Gabriel Charpentier. The documentation was initially autogenerated by the SentenceTransformers library then revised by Victoria Handford, Lucas Georges Gabriel Charpentier, and Javier de la Rosa.
