This is a sentence-transformers model finetuned from microsoft/mpnet-base on the sentence-transformers/gooaq dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model was trained using the train_script.py code.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: microsoft/mpnet-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity
Training Dataset:
- sentence-transformers/gooaq
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/mpnet-base-gooaq")
# Run inference
sentences = [
    '11 is what of 8?',
    'Convert fraction (ratio) 8 / 11 Answer: 72.727272727273%',
    'Old-age pensions are not included in taxable income under the personal income tax.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Dataset: gooaq-dev
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7198
cosine_accuracy@3	0.884
cosine_accuracy@5	0.9305
cosine_accuracy@10	0.9709
cosine_precision@1	0.7198
cosine_precision@3	0.2947
cosine_precision@5	0.1861
cosine_precision@10	0.0971
cosine_recall@1	0.7198
cosine_recall@3	0.884
cosine_recall@5	0.9305
cosine_recall@10	0.9709
cosine_ndcg@10	0.8491
cosine_mrr@10	0.8096
cosine_map@100	0.8111
dot_accuracy@1	0.7073
dot_accuracy@3	0.877
dot_accuracy@5	0.9244
dot_accuracy@10	0.9669
dot_precision@1	0.7073
dot_precision@3	0.2923
dot_precision@5	0.1849
dot_precision@10	0.0967
dot_recall@1	0.7073
dot_recall@3	0.877
dot_recall@5	0.9244
dot_recall@10	0.9669
dot_ndcg@10	0.8412
dot_mrr@10	0.8004
dot_map@100	0.8023

Training Details

Training Dataset

sentence-transformers/gooaq

Dataset: sentence-transformers/gooaq at b089f72
Size: 3,002,496 training samples
Columns: question and answer
Approximate statistics based on the first 1000 samples:
question answer
type string string
details
min: 8 tokens
mean: 11.89 tokens
max: 22 tokens
min: 15 tokens
mean: 60.37 tokens
max: 147 tokens

Samples:

question	answer
`biotechnology is best defined as?`	`Biotechnology is best defined as_______________? The science that involves using living organisms to produce needed materials. Which of the following tools of biotechnology, to do investigation, is used when trying crime?`
`how to open xye file?`	`Firstly, use File then Open and make sure that you can see All Files (. ) and not just Excel files (the default option!) in the folder containing the *. xye file: Select the file you wish to open and Excel will bring up a wizard menu for importing plain text data into Excel (as shown below).`
`how much does california spend?`	`Estimated 2016 expenditures The total estimated government spending in California in fiscal year 2016 was $265.9 billion. Per-capita figures are calculated by taking the state's total spending and dividing by the number of state residents according to United States Census Bureau estimates.`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Evaluation Dataset

sentence-transformers/gooaq

Dataset: sentence-transformers/gooaq at b089f72
Size: 10,000 evaluation samples
Columns: question and answer
Approximate statistics based on the first 1000 samples:
question answer
type string string
details
min: 8 tokens
mean: 11.86 tokens
max: 25 tokens
min: 14 tokens
mean: 60.82 tokens
max: 166 tokens

Samples:

question	answer
`how to open nx file?`	`['Click File > Open. The File Open dialog box opens.', 'Select NX File (*. prt) in the Type box. ... ', 'Select an NX . ... ', 'Select Import in the File Open dialog box. ... ', 'If you do not want to retain the import profile in use, select an import profile from the Profile list. ... ', 'Click OK in the Import New Model dialog box.']`
`how to recover deleted photos from blackberry priv?`	`['Run Android Data Recovery. ... ', 'Enable USB Debugging Mode. ... ', 'Scan Your BlackBerry PRIV to Find Deleted Photos. ... ', 'Recover Deleted Photos from BlackBerry PRIV.']`
`which subatomic particles are found within the nucleus of an atom?`	`In the middle of every atom is the nucleus. The nucleus contains two types of subatomic particles, protons and neutrons. The protons have a positive electrical charge and the neutrons have no electrical charge. A third type of subatomic particle, electrons, move around the nucleus.`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 64
per_device_eval_batch_size: 64
learning_rate: 2e-05
num_train_epochs: 1
warmup_ratio: 0.1
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 64
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	loss	gooaq-dev_cosine_map@100
0	0	-	-	0.1379
0.0000	1	3.6452	-	-
0.0053	250	2.4418	-	-
0.0107	500	0.373	-	-
0.0160	750	0.183	-	-
0.0213	1000	0.1286	0.0805	0.6796
0.0266	1250	0.1099	-	-
0.0320	1500	0.091	-	-
0.0373	1750	0.0768	-	-
0.0426	2000	0.0665	0.0526	0.7162
0.0480	2250	0.0659	-	-
0.0533	2500	0.0602	-	-
0.0586	2750	0.0548	-	-
0.0639	3000	0.0543	0.0426	0.7328
0.0693	3250	0.0523	-	-
0.0746	3500	0.0494	-	-
0.0799	3750	0.0468	-	-
0.0853	4000	0.0494	0.0362	0.7450
0.0906	4250	0.048	-	-
0.0959	4500	0.0442	-	-
0.1012	4750	0.0442	-	-
0.1066	5000	0.0408	0.0332	0.7519
0.1119	5250	0.0396	-	-
0.1172	5500	0.0379	-	-
0.1226	5750	0.0392	-	-
0.1279	6000	0.0395	0.0300	0.7505
0.1332	6250	0.0349	-	-
0.1386	6500	0.0383	-	-
0.1439	6750	0.0335	-	-
0.1492	7000	0.0323	0.0253	0.7624
0.1545	7250	0.0342	-	-
0.1599	7500	0.0292	-	-
0.1652	7750	0.0309	-	-
0.1705	8000	0.0335	0.0249	0.7631
0.1759	8250	0.0304	-	-
0.1812	8500	0.0318	-	-
0.1865	8750	0.0271	-	-
0.1918	9000	0.029	0.0230	0.7615
0.1972	9250	0.0309	-	-
0.2025	9500	0.0305	-	-
0.2078	9750	0.0237	-	-
0.2132	10000	0.0274	0.0220	0.7667
0.2185	10250	0.0248	-	-
0.2238	10500	0.0249	-	-
0.2291	10750	0.0272	-	-
0.2345	11000	0.0289	0.0230	0.7664
0.2398	11250	0.027	-	-
0.2451	11500	0.0259	-	-
0.2505	11750	0.0237	-	-
0.2558	12000	0.0245	0.0220	0.7694
0.2611	12250	0.0251	-	-
0.2664	12500	0.0243	-	-
0.2718	12750	0.0229	-	-
0.2771	13000	0.0273	0.0201	0.7725
0.2824	13250	0.0244	-	-
0.2878	13500	0.0248	-	-
0.2931	13750	0.0255	-	-
0.2984	14000	0.0244	0.0192	0.7729
0.3037	14250	0.0242	-	-
0.3091	14500	0.0235	-	-
0.3144	14750	0.0231	-	-
0.3197	15000	0.0228	0.0190	0.7823
0.3251	15250	0.0229	-	-
0.3304	15500	0.0224	-	-
0.3357	15750	0.0216	-	-
0.3410	16000	0.0218	0.0186	0.7787
0.3464	16250	0.022	-	-
0.3517	16500	0.0233	-	-
0.3570	16750	0.0216	-	-
0.3624	17000	0.0226	0.0169	0.7862
0.3677	17250	0.0215	-	-
0.3730	17500	0.0212	-	-
0.3784	17750	0.0178	-	-
0.3837	18000	0.0217	0.0161	0.7813
0.3890	18250	0.0217	-	-
0.3943	18500	0.0191	-	-
0.3997	18750	0.0216	-	-
0.4050	19000	0.022	0.0157	0.7868
0.4103	19250	0.0223	-	-
0.4157	19500	0.021	-	-
0.4210	19750	0.0176	-	-
0.4263	20000	0.021	0.0162	0.7873
0.4316	20250	0.0206	-	-
0.4370	20500	0.0196	-	-
0.4423	20750	0.0186	-	-
0.4476	21000	0.0197	0.0158	0.7907
0.4530	21250	0.0156	-	-
0.4583	21500	0.0178	-	-
0.4636	21750	0.0175	-	-
0.4689	22000	0.0187	0.0151	0.7937
0.4743	22250	0.0182	-	-
0.4796	22500	0.0185	-	-
0.4849	22750	0.0217	-	-
0.4903	23000	0.0179	0.0156	0.7937
0.4956	23250	0.0193	-	-
0.5009	23500	0.015	-	-
0.5062	23750	0.0181	-	-
0.5116	24000	0.0173	0.0150	0.7924
0.5169	24250	0.0177	-	-
0.5222	24500	0.0183	-	-
0.5276	24750	0.0171	-	-
0.5329	25000	0.0185	0.0140	0.7955
0.5382	25250	0.0178	-	-
0.5435	25500	0.015	-	-
0.5489	25750	0.017	-	-
0.5542	26000	0.0171	0.0139	0.7931
0.5595	26250	0.0164	-	-
0.5649	26500	0.0175	-	-
0.5702	26750	0.0175	-	-
0.5755	27000	0.0163	0.0133	0.7954
0.5809	27250	0.0179	-	-
0.5862	27500	0.016	-	-
0.5915	27750	0.0155	-	-
0.5968	28000	0.0162	0.0138	0.7979
0.6022	28250	0.0164	-	-
0.6075	28500	0.0148	-	-
0.6128	28750	0.0152	-	-
0.6182	29000	0.0166	0.0134	0.7987
0.6235	29250	0.0159	-	-
0.6288	29500	0.0168	-	-
0.6341	29750	0.0187	-	-
0.6395	30000	0.017	0.0137	0.7980
0.6448	30250	0.0168	-	-
0.6501	30500	0.0149	-	-
0.6555	30750	0.0159	-	-
0.6608	31000	0.0149	0.0131	0.8017
0.6661	31250	0.0149	-	-
0.6714	31500	0.0147	-	-
0.6768	31750	0.0157	-	-
0.6821	32000	0.0151	0.0125	0.8011
0.6874	32250	0.015	-	-
0.6928	32500	0.0157	-	-
0.6981	32750	0.0153	-	-
0.7034	33000	0.0141	0.0123	0.8012
0.7087	33250	0.0143	-	-
0.7141	33500	0.0121	-	-
0.7194	33750	0.0164	-	-
0.7247	34000	0.014	0.0121	0.8014
0.7301	34250	0.0147	-	-
0.7354	34500	0.0149	-	-
0.7407	34750	0.014	-	-
0.7460	35000	0.0156	0.0117	0.8022
0.7514	35250	0.0153	-	-
0.7567	35500	0.0146	-	-
0.7620	35750	0.0144	-	-
0.7674	36000	0.0139	0.0111	0.8035
0.7727	36250	0.0134	-	-
0.7780	36500	0.013	-	-
0.7833	36750	0.0156	-	-
0.7887	37000	0.0144	0.0108	0.8048
0.7940	37250	0.0133	-	-
0.7993	37500	0.0154	-	-
0.8047	37750	0.0132	-	-
0.8100	38000	0.013	0.0108	0.8063
0.8153	38250	0.0126	-	-
0.8207	38500	0.0135	-	-
0.8260	38750	0.014	-	-
0.8313	39000	0.013	0.0109	0.8086
0.8366	39250	0.0136	-	-
0.8420	39500	0.0141	-	-
0.8473	39750	0.0155	-	-
0.8526	40000	0.0153	0.0106	0.8075
0.8580	40250	0.0131	-	-
0.8633	40500	0.0128	-	-
0.8686	40750	0.013	-	-
0.8739	41000	0.0133	0.0109	0.8060
0.8793	41250	0.0119	-	-
0.8846	41500	0.0144	-	-
0.8899	41750	0.0142	-	-
0.8953	42000	0.0138	0.0105	0.8083
0.9006	42250	0.014	-	-
0.9059	42500	0.0134	-	-
0.9112	42750	0.0134	-	-
0.9166	43000	0.0124	0.0106	0.8113
0.9219	43250	0.0122	-	-
0.9272	43500	0.0126	-	-
0.9326	43750	0.0121	-	-
0.9379	44000	0.0137	0.0103	0.8105
0.9432	44250	0.0132	-	-
0.9485	44500	0.012	-	-
0.9539	44750	0.0136	-	-
0.9592	45000	0.0133	0.0104	0.8112
0.9645	45250	0.0118	-	-
0.9699	45500	0.0132	-	-
0.9752	45750	0.0118	-	-
0.9805	46000	0.012	0.0102	0.8104
0.9858	46250	0.0127	-	-
0.9912	46500	0.0134	-	-
0.9965	46750	0.0121	-	-
1.0	46914	-	-	0.8111

Environmental Impact

Carbon emissions were measured using CodeCarbon.

Energy Consumed: 1.637 kWh
Carbon Emitted: 0.636 kg of CO2
Hours Used: 4.514 hours

Training Hardware

On Cloud: No
GPU Model: 1 x NVIDIA GeForce RTX 3090
CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
RAM Size: 31.78 GB

Framework Versions

Python: 3.11.6
Sentence Transformers: 3.1.0.dev0
Transformers: 4.41.2
PyTorch: 2.3.0+cu121
Accelerate: 0.30.1
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	question	answer
type	string	string
details	min: 8 tokens mean: 11.89 tokens max: 22 tokens	min: 15 tokens mean: 60.37 tokens max: 147 tokens