This is a sentence-transformers model trained on the msmarco-embed-dense_on-triples-v3 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

Model Type: Sentence Transformer

Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text
Training Dataset:
- msmarco-embed-dense_on-triples-v3

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 256, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 256, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'define reservedness',
]
documents = [
    'Definition of reservedness in the English dictionary The definition of reservedness in the dictionary is the quality or state of being cool or formal in manner or of being restrained, silent, or reticent. english words that begin like reservedness reserve tank',
    "Princeton's WordNet(0.00 / 0 votes)Rate this definition: 1  reserved(adj) set aside for the use of a particular person or party. 2  reserved(adj) marked by self-restraint and reticence.",
    'In a computer language, a reserved word is a word that cannot be used as an identifier, such as the name of a variable, function, or label â\x80\x93 it is reserved from use. This is a syntactic definition, and a reserved word may have no meaning. A closely related and often conflated notion is a keyword, which is a word with special meaning in a particular context. This is a semantic definition. By contrast, names in a standard library but not built into the language are not considered reserved ...',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5359, 0.4371, 0.3782]])

Evaluation

Metrics

Information Retrieval

Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with InformationRetrievalEvaluator with these parameters:
json
1{ 2 "query_prompt": "query: ", 3 "corpus_prompt": "document: " 4}

Metric	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
cosine_accuracy@1	0.28	0.68	0.46	0.38	0.56	0.36	0.38	0.28	0.9	0.28	0.08	0.5	0.5918
cosine_accuracy@3	0.42	0.8	0.6	0.56	0.74	0.54	0.5	0.58	0.94	0.44	0.44	0.58	0.8571
cosine_accuracy@5	0.5	0.84	0.74	0.6	0.8	0.66	0.56	0.6	0.98	0.52	0.5	0.6	0.8571
cosine_accuracy@10	0.62	0.86	0.88	0.72	0.88	0.76	0.6	0.7	1.0	0.62	0.64	0.68	0.9592
cosine_precision@1	0.28	0.68	0.46	0.38	0.56	0.36	0.38	0.28	0.9	0.28	0.08	0.5	0.5918
cosine_precision@3	0.1667	0.54	0.2067	0.2333	0.3333	0.18	0.3467	0.1933	0.3733	0.2067	0.1467	0.2133	0.5374
cosine_precision@5	0.124	0.488	0.152	0.164	0.22	0.132	0.288	0.12	0.248	0.176	0.1	0.132	0.4939
cosine_precision@10	0.076	0.44	0.09	0.102	0.134	0.076	0.246	0.072	0.134	0.118	0.064	0.076	0.402
cosine_recall@1	0.1167	0.0716	0.45	0.2155	0.28	0.36	0.0203	0.26	0.7873	0.0597	0.08	0.465	0.0412
cosine_recall@3	0.2083	0.1191	0.59	0.3522	0.5	0.54	0.0518	0.54	0.8887	0.1297	0.44	0.57	0.1115
cosine_recall@5	0.2667	0.1695	0.7167	0.3986	0.55	0.66	0.0609	0.56	0.9453	0.1817	0.5	0.59	0.1729
cosine_recall@10	0.3223	0.249	0.8467	0.5015	0.67	0.76	0.1071	0.66	0.99	0.2427	0.64	0.675	0.2696
cosine_ndcg@10	0.2664	0.5252	0.6348	0.4125	0.5719	0.5486	0.2962	0.4793	0.9318	0.2374	0.3685	0.5761	0.4614
cosine_mrr@10	0.3735	0.75	0.5736	0.4898	0.6602	0.482	0.4575	0.4385	0.9325	0.388	0.281	0.5521	0.7255
cosine_map@100	0.2114	0.3713	0.5679	0.341	0.4906	0.4925	0.1066	0.4245	0.9017	0.1813	0.2906	0.5515	0.357

Nano BEIR

Dataset: NanoBEIR_mean
Evaluated with NanoBEIREvaluator with these parameters:
json
1{ 2 "dataset_names": [ 3 "climatefever", 4 "dbpedia", 5 "fever", 6 "fiqa2018", 7 "hotpotqa", 8 "msmarco", 9 "nfcorpus", 10 "nq", 11 "quoraretrieval", 12 "scidocs", 13 "arguana", 14 "scifact", 15 "touche2020" 16 ], 17 "dataset_id": "sentence-transformers/NanoBEIR-en", 18 "query_prompts": { 19 "climatefever": "query: ", 20 "dbpedia": "query: ", 21 "fever": "query: ", 22 "fiqa2018": "query: ", 23 "hotpotqa": "query: ", 24 "msmarco": "query: ", 25 "nfcorpus": "query: ", 26 "nq": "query: ", 27 "quoraretrieval": "query: ", 28 "scidocs": "query: ", 29 "arguana": "query: ", 30 "scifact": "query: ", 31 "touche2020": "query: " 32 }, 33 "corpus_prompts": { 34 "climatefever": "document: ", 35 "dbpedia": "document: ", 36 "fever": "document: ", 37 "fiqa2018": "document: ", 38 "hotpotqa": "document: ", 39 "msmarco": "document: ", 40 "nfcorpus": "document: ", 41 "nq": "document: ", 42 "quoraretrieval": "document: ", 43 "scidocs": "document: ", 44 "arguana": "document: ", 45 "scifact": "document: ", 46 "touche2020": "document: " 47 } 48}

Metric	Value
cosine_accuracy@1	0.4409
cosine_accuracy@3	0.6152
cosine_accuracy@5	0.6736
cosine_accuracy@10	0.763
cosine_precision@1	0.4409
cosine_precision@3	0.2829
cosine_precision@5	0.2183
cosine_precision@10	0.1562
cosine_recall@1	0.2467
cosine_recall@3	0.3878
cosine_recall@5	0.444
cosine_recall@10	0.5334
cosine_ndcg@10	0.4854
cosine_mrr@10	0.5465
cosine_map@100	0.4068

Training Details

Training Dataset

msmarco-embed-dense_on-triples-v3

Dataset: msmarco-embed-dense_on-triples-v3
Size: 483,497 training samples
Columns: query, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, and label

Approximate statistics based on the first 100 samples:

	query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	label
type	string	string	string	string	string	string	string	string	string	string	list
modality	text	text	text	text	text	text	text	text	text	text
details	min: 5 tokens mean: 9.21 tokens max: 21 tokens	min: 28 tokens mean: 79.34 tokens max: 201 tokens	min: 16 tokens mean: 67.1 tokens max: 167 tokens	min: 21 tokens mean: 69.42 tokens max: 159 tokens	min: 15 tokens mean: 68.48 tokens max: 138 tokens	min: 22 tokens mean: 70.42 tokens max: 179 tokens	min: 16 tokens mean: 70.38 tokens max: 226 tokens	min: 22 tokens mean: 70.48 tokens max: 183 tokens	min: 18 tokens mean: 73.81 tokens max: 202 tokens	min: 26 tokens mean: 71.48 tokens max: 162 tokens	size: 10 elements

Loss: EmbedDistillLoss with these parameters:
json
1{ 2 "distance_metric": "l2", 3 "projection_dim": null 4}

Evaluation Dataset

msmarco-embed-dense_on-triples-v3

Dataset: msmarco-embed-dense_on-triples-v3
Size: 10,000 evaluation samples
Columns: query, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, and label

Approximate statistics based on the first 100 samples:

	query	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	label
type	string	string	string	string	string	string	string	string	string	string	list
modality	text	text	text	text	text	text	text	text	text	text
details	min: 5 tokens mean: 9.09 tokens max: 19 tokens	min: 31 tokens mean: 83.38 tokens max: 226 tokens	min: 21 tokens mean: 74.75 tokens max: 167 tokens	min: 21 tokens mean: 65.31 tokens max: 175 tokens	min: 22 tokens mean: 74.03 tokens max: 223 tokens	min: 18 tokens mean: 77.41 tokens max: 192 tokens	min: 18 tokens mean: 74.96 tokens max: 214 tokens	min: 23 tokens mean: 70.17 tokens max: 136 tokens	min: 22 tokens mean: 72.93 tokens max: 170 tokens	min: 11 tokens mean: 72.15 tokens max: 194 tokens	size: 10 elements

Loss: EmbedDistillLoss with these parameters:

json
1{ 2 "distance_metric": "l2", 3 "projection_dim": null 4}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
num_train_epochs: 2
max_steps: 30000
learning_rate: 0.0001
lr_scheduler_type: constant
weight_decay: 0.01
bf16: True
per_device_eval_batch_size: 16
seed: 12
dataloader_num_workers: 2
remove_unused_columns: False

All Hyperparameters

Click to expand

per_device_train_batch_size: 32
num_train_epochs: 2
max_steps: 30000
learning_rate: 0.0001
lr_scheduler_type: constant
lr_scheduler_kwargs: None
warmup_steps: 0
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 16
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 12
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 2
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: False
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoClimateFEVER_cosine_ndcg@10	NanoDBPedia_cosine_ndcg@10	NanoFEVER_cosine_ndcg@10	NanoFiQA2018_cosine_ndcg@10	NanoHotpotQA_cosine_ndcg@10	NanoMSMARCO_cosine_ndcg@10	NanoNFCorpus_cosine_ndcg@10	NanoNQ_cosine_ndcg@10	NanoQuoraRetrieval_cosine_ndcg@10	NanoSCIDOCS_cosine_ndcg@10	NanoArguAna_cosine_ndcg@10	NanoSciFact_cosine_ndcg@10	NanoTouche2020_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10
-1	-1	-	-	0.1218	0.3308	0.2241	0.2243	0.3777	0.3582	0.1359	0.2804	0.8769	0.1972	0.2998	0.4503	0.2692	0.3190
0.0001	1	1.1693	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0331	500	0.6371	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0662	1000	0.6070	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0993	1500	0.6016	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1324	2000	0.5972	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1655	2500	0.5930	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1985	3000	0.5911	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2316	3500	0.5877	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2647	4000	0.5869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2978	4500	0.5846	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3309	5000	0.5818	0.5812	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3640	5500	0.5815	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3971	6000	0.5784	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4302	6500	0.5782	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4633	7000	0.5755	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4964	7500	0.5741	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5295	8000	0.5735	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5625	8500	0.5728	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5956	9000	0.5705	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6287	9500	0.5695	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6618	10000	0.5701	0.5694	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6949	10500	0.5786	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7280	11000	0.5768	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7611	11500	0.5741	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7942	12000	0.5716	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8273	12500	0.5718	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8604	13000	0.5686	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8934	13500	0.5673	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9265	14000	0.5658	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9596	14500	0.5654	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9927	15000	0.5646	0.5628	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0258	15500	0.5643	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0589	16000	0.5601	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0920	16500	0.5581	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.1251	17000	0.5579	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.1582	17500	0.5558	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.1913	18000	0.5564	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.2244	18500	0.5548	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.2574	19000	0.5553	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.2905	19500	0.5532	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.3236	20000	0.5549	0.5544	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.3567	20500	0.5617	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.3898	21000	0.5602	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.4229	21500	0.5598	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.4560	22000	0.5589	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.4891	22500	0.5569	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.5222	23000	0.5545	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.5553	23500	0.5545	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.5884	24000	0.5517	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.6214	24500	0.5525	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.6545	25000	0.5524	0.5513	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.6876	25500	0.5517	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.7207	26000	0.5515	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.7538	26500	0.5488	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.7869	27000	0.5479	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.8200	27500	0.5469	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.8531	28000	0.5463	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.8862	28500	0.5463	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.9193	29000	0.5437	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.9523	29500	0.5445	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.9854	30000	0.5452	0.5442	-	-	-	-	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.2664	0.5252	0.6348	0.4125	0.5719	0.5486	0.2962	0.4793	0.9318	0.2374	0.3685	0.5761	0.4614	0.4854

Training Time

Training: 1.8 hours
Evaluation: 8.0 minutes
Total: 1.9 hours

Framework Versions

Python: 3.11.12
Sentence Transformers: 5.6.1
Transformers: 5.14.1
PyTorch: 2.13.0+cu130
Accelerate: 1.14.0
Datasets: 5.0.1
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

EmbedDistillLoss

bibtex

@article{kim2023embeddistill,
    title={EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval},
    author={Kim, Seungyeon and Rawat, Ankit Singh and Zaheer, Manzil and Jayasumana, Sadeep and Sadhanala, Veeranjaneyulu and Jitkrittum, Wittawat and Menon, Aditya Krishna and Fergus, Rob and Kumar, Sanjiv},
    year={2023},
    eprint={2301.12005},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}