This is a sentence-transformers model finetuned from sentence-transformers/multi-qa-MiniLM-L6-cos-v1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/multi-qa-MiniLM-L6-cos-v1
Maximum Sequence Length: 512 tokens
Output Dimensionality: 384 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Bone Saw',
    'Bone Saw Sklar  Inch',
    'Mask Component  Headgear Opus',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 231,882 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 3 tokens
mean: 14.16 tokens
max: 49 tokens
min: 3 tokens
mean: 13.28 tokens
max: 53 tokens

Samples:

anchor	positive
`Biopsy Cassette Thermo Scientific Shandon Acetal Blue`	`Biopsy Cassette Blue Acetal`
`Tissue Cassette Thermo Scientific Shandon Acetal Fluorescent Green`	`Tissue Cassette Fluorescent Green Acetal`
`Tissue Cassette Thermo Scientific Shandon Acetal Fluorescent Pink`	`Tissue Cassette Fluorescent Pink Acetal`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

num_train_epochs: 4
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0172	500	0.1383
0.0345	1000	0.1183
0.0517	1500	0.1054
0.0690	2000	0.0727
0.0862	2500	0.0829
0.1035	3000	0.0559
0.1207	3500	0.1274
0.1380	4000	0.0587
0.1552	4500	0.0704
0.1725	5000	0.0863
0.1897	5500	0.0888
0.2070	6000	0.1099
0.2242	6500	0.1126
0.2415	7000	0.1192
0.2587	7500	0.1082
0.2760	8000	0.1069
0.2932	8500	0.1268
0.3105	9000	0.0913
0.3277	9500	0.1267
0.3450	10000	0.1156
0.3622	10500	0.1522
0.3795	11000	0.088
0.3967	11500	0.0906
0.4140	12000	0.0776
0.4312	12500	0.0956
0.4485	13000	0.1111
0.4657	13500	0.0889
0.4830	14000	0.0765
0.5002	14500	0.1162
0.5175	15000	0.0581
0.5347	15500	0.0831
0.5520	16000	0.0915
0.5692	16500	0.0623
0.5865	17000	0.0702
0.6037	17500	0.0447
0.6210	18000	0.0715
0.6382	18500	0.0749
0.6555	19000	0.3381
0.6727	19500	0.0749
0.6900	20000	0.0614
0.7072	20500	0.1093
0.7245	21000	0.0847
0.7417	21500	0.063
0.7590	22000	0.0657
0.7762	22500	0.061
0.7935	23000	0.0837
0.8107	23500	0.0989
0.8280	24000	0.0523
0.8452	24500	0.0817
0.8625	25000	0.0533
0.8797	25500	0.0584
0.8970	26000	0.0353
0.9142	26500	0.0146
0.9315	27000	0.0831
0.9487	27500	0.049
0.9660	28000	0.0741
0.9832	28500	0.0469
1.0004	29000	0.063
1.0177	29500	0.0846
1.0349	30000	0.058
1.0522	30500	0.0701
1.0694	31000	0.0451
1.0867	31500	0.0506
1.1039	32000	0.0311
1.1212	32500	0.0761
1.1384	33000	0.0356
1.1557	33500	0.0387
1.1729	34000	0.0532
1.1902	34500	0.0568
1.2074	35000	0.0654
1.2247	35500	0.0726
1.2419	36000	0.0839
1.2592	36500	0.0698
1.2764	37000	0.0824
1.2937	37500	0.0832
1.3109	38000	0.0622
1.3282	38500	0.0849
1.3454	39000	0.0724
1.3627	39500	0.1039
1.3799	40000	0.0581
1.3972	40500	0.0561
1.4144	41000	0.0666
1.4317	41500	0.0687
1.4489	42000	0.0793
1.4662	42500	0.0638
1.4834	43000	0.0544
1.5007	43500	0.0686
1.5179	44000	0.0408
1.5352	44500	0.0602
1.5524	45000	0.0663
1.5697	45500	0.0488
1.5869	46000	0.047
1.6042	46500	0.0326
1.6214	47000	0.0644
1.6387	47500	0.0582
1.6559	48000	0.2124
1.6732	48500	0.0482
1.6904	49000	0.0389
1.7077	49500	0.0847
1.7249	50000	0.0636
1.7422	50500	0.044
1.7594	51000	0.0403
1.7767	51500	0.0397
1.7939	52000	0.0545
1.8112	52500	0.0681
1.8284	53000	0.0422
1.8456	53500	0.0522
1.8629	54000	0.0394
1.8801	54500	0.041
1.8974	55000	0.0232
1.9146	55500	0.0176
1.9319	56000	0.0471
1.9491	56500	0.0337
1.9664	57000	0.0439
1.9836	57500	0.0321
2.0008	58000	0.0433
2.0181	58500	0.0672
2.0353	59000	0.0441
2.0526	59500	0.0459
2.0698	60000	0.0342
2.0871	60500	0.0369
2.1043	61000	0.0205
2.1216	61500	0.0605
2.1388	62000	0.0252
2.1561	62500	0.0276
2.1733	63000	0.0406
2.1906	63500	0.0451
2.2078	64000	0.0447
2.2251	64500	0.0523
2.2423	65000	0.062
2.2596	65500	0.0514
2.2768	66000	0.0677
2.2941	66500	0.0655
2.3113	67000	0.0494
2.3286	67500	0.0728
2.3458	68000	0.0585
2.3631	68500	0.0866
2.3803	69000	0.0409
2.3976	69500	0.0429
2.4148	70000	0.0534
2.4321	70500	0.0542
2.4493	71000	0.0563
2.4666	71500	0.0488
2.4838	72000	0.0401
2.5011	72500	0.0575
2.5183	73000	0.0344
2.5356	73500	0.052
2.5528	74000	0.0569
2.5701	74500	0.0408
2.5873	75000	0.0384
2.6046	75500	0.0281
2.6218	76000	0.0447
2.6391	76500	0.0495
2.6563	77000	0.1492
2.6736	77500	0.0314
2.6908	78000	0.0314
2.7081	78500	0.0691
2.7253	79000	0.0496
2.7426	79500	0.0309
2.7598	80000	0.0323
2.7771	80500	0.0357
2.7943	81000	0.0387
2.8116	81500	0.0544
2.8288	82000	0.0297
2.8461	82500	0.0384
2.8633	83000	0.0332
2.8806	83500	0.031
2.8978	84000	0.017
2.9151	84500	0.0223
2.9323	85000	0.0271
2.9496	85500	0.0298
2.9668	86000	0.0297
2.9841	86500	0.026
3.0012	87000	0.0266
3.0185	87500	0.0531
3.0357	88000	0.0342
3.0530	88500	0.039
3.0702	89000	0.0263
3.0875	89500	0.0288
3.1047	90000	0.0158
3.1220	90500	0.0484
3.1392	91000	0.0179
3.1565	91500	0.0215
3.1737	92000	0.0316
3.1910	92500	0.0395
3.2082	93000	0.037
3.2255	93500	0.0389
3.2427	94000	0.0512
3.2600	94500	0.0451
3.2772	95000	0.0583
3.2945	95500	0.0502
3.3117	96000	0.0407
3.3290	96500	0.0628
3.3462	97000	0.0434
3.3635	97500	0.0741
3.3807	98000	0.0318
3.3980	98500	0.0387
3.4152	99000	0.041
3.4325	99500	0.0429
3.4497	100000	0.0514
3.4670	100500	0.0377
3.4842	101000	0.0355
3.5015	101500	0.043
3.5187	102000	0.029
3.5360	102500	0.047
3.5532	103000	0.0554
3.5705	103500	0.0385
3.5877	104000	0.0294
3.6050	104500	0.023
3.6222	105000	0.0381
3.6395	105500	0.0422
3.6567	106000	0.1091
3.6740	106500	0.0289
3.6912	107000	0.0276
3.7085	107500	0.0606
3.7257	108000	0.0402
3.7430	108500	0.0256
3.7602	109000	0.0279
3.7775	109500	0.0317
3.7947	110000	0.0303
3.8120	110500	0.0492
3.8292	111000	0.0239
3.8465	111500	0.0297
3.8637	112000	0.0293
3.8810	112500	0.0278
3.8982	113000	0.0134
3.9155	113500	0.0192
3.9327	114000	0.0235
3.9500	114500	0.0268
3.9672	115000	0.022
3.9845	115500	0.0235

Framework Versions

Python: 3.9.19
Sentence Transformers: 3.0.0
Transformers: 4.41.2
PyTorch: 2.3.0+cu121
Accelerate: 0.30.1
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss