This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the parquet dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-MiniLM-L6-v2
Maximum Sequence Length: 256 tokens
Output Dimensionality: 384 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- parquet

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("yyzheng00/snomed_triplet_500k")
# Run inference
sentences = [
    '|Structure of right side of trunk| + |Structure of soft tissue of back of thoracic segment of trunk| : |Laterality| = |Right|, ',
    'Structure of soft tissue of right half of back of thoracic segment of trunk',
    'Structure of vein of right limb',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Dataset: snomed_triplet_500k_3_4_3-dev
Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9986

Triplet

Dataset: snomed_triplet_500k_3_4_3-dev
Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9987

Training Details

Training Dataset

parquet

Dataset: parquet
Size: 500,000 training samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 1000 samples:
anchor positive negative
type string string string
details
min: 18 tokens
mean: 61.91 tokens
max: 247 tokens
min: 4 tokens
mean: 11.04 tokens
max: 39 tokens
min: 4 tokens
mean: 11.29 tokens
max: 50 tokens
Samples:
anchor positive negative
Supernumerary deciduous mandibular tooth : {
Hemorrhage into subdural space of neuraxis +
Deep mammalian bite wound +
Loss: TripletLoss with these parameters:
json
1{ 2 "distance_metric": "TripletDistanceMetric.COSINE", 3 "triplet_margin": 0.2 4}

Evaluation Dataset

parquet

Dataset: parquet
Size: 500,000 evaluation samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 1000 samples:
anchor positive negative
type string string string
details
min: 18 tokens
mean: 62.56 tokens
max: 256 tokens
min: 3 tokens
mean: 10.81 tokens
max: 33 tokens
min: 3 tokens
mean: 11.32 tokens
max: 60 tokens
Samples:
anchor positive negative
Anastomosis of duodenum to colon +
Benign neoplasm of colon +
Closed traumatic dislocation of hip +
Loss: TripletLoss with these parameters:
json
1{ 2 "distance_metric": "TripletDistanceMetric.COSINE", 3 "triplet_margin": 0.2 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
num_train_epochs: 1
warmup_ratio: 0.1
fp16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	snomed_triplet_500k_3_4_3-dev_cosine_accuracy
0.0053	100	0.0143	0.0084	0.9899
0.0107	200	0.009	0.0066	0.9918
0.016	300	0.0082	0.0057	0.9925
0.0213	400	0.0066	0.0053	0.993
0.0267	500	0.0054	0.0049	0.9936
0.032	600	0.0059	0.0047	0.9939
0.0373	700	0.0053	0.0044	0.9946
0.0427	800	0.0047	0.0042	0.9945
0.048	900	0.0048	0.0040	0.9947
0.0533	1000	0.0046	0.0040	0.9951
0.0587	1100	0.0048	0.0039	0.9954
0.064	1200	0.0057	0.0038	0.9948
0.0693	1300	0.0049	0.0038	0.9952
0.0747	1400	0.0038	0.0042	0.9944
0.08	1500	0.0035	0.0038	0.9953
0.0853	1600	0.0034	0.0034	0.9962
0.0907	1700	0.0044	0.0036	0.9957
0.096	1800	0.0041	0.0038	0.9950
0.1013	1900	0.0042	0.0040	0.9948
0.1067	2000	0.0052	0.0037	0.9952
0.112	2100	0.0045	0.0038	0.9953
0.1173	2200	0.0054	0.0034	0.9961
0.1227	2300	0.0041	0.0034	0.9956
0.128	2400	0.0053	0.0035	0.9954
0.1333	2500	0.0043	0.0035	0.9957
0.1387	2600	0.0046	0.0034	0.9955
0.144	2700	0.0039	0.0033	0.9959
0.1493	2800	0.0039	0.0034	0.9958
0.1547	2900	0.0041	0.0033	0.9959
0.16	3000	0.0036	0.0033	0.9960
0.1653	3100	0.0046	0.0033	0.9960
0.1707	3200	0.0041	0.0033	0.9958
0.176	3300	0.0043	0.0033	0.9959
0.1813	3400	0.0041	0.0032	0.9958
0.1867	3500	0.0042	0.0031	0.9961
0.192	3600	0.0049	0.0029	0.9963
0.1973	3700	0.0035	0.0030	0.9960
0.2027	3800	0.0042	0.0029	0.9963
0.208	3900	0.0034	0.0028	0.9966
0.2133	4000	0.0035	0.0030	0.9959
0.2187	4100	0.0045	0.0028	0.9966
0.224	4200	0.0041	0.0028	0.9965
0.2293	4300	0.0027	0.0027	0.9965
0.2347	4400	0.0031	0.0027	0.9967
0.24	4500	0.0039	0.0027	0.9967
0.2453	4600	0.0051	0.0026	0.9972
0.2507	4700	0.0027	0.0029	0.9963
0.256	4800	0.0034	0.0027	0.9968
0.2613	4900	0.003	0.0025	0.9969
0.2667	5000	0.003	0.0025	0.9972
0.272	5100	0.0023	0.0026	0.9971
0.2773	5200	0.0038	0.0026	0.9967
0.2827	5300	0.0027	0.0026	0.9968
0.288	5400	0.0039	0.0029	0.9973
0.2933	5500	0.0031	0.0027	0.9969
0.2987	5600	0.0022	0.0025	0.9970
0.304	5700	0.0024	0.0025	0.9972
0.3093	5800	0.0032	0.0025	0.9968
0.3147	5900	0.0035	0.0025	0.9968
0.32	6000	0.002	0.0025	0.9972
0.3253	6100	0.0025	0.0026	0.9970
0.3307	6200	0.003	0.0026	0.9972
0.336	6300	0.0032	0.0027	0.9970
0.3413	6400	0.0033	0.0026	0.997
0.3467	6500	0.0026	0.0024	0.9973
0.352	6600	0.0029	0.0024	0.9972
0.3573	6700	0.002	0.0024	0.9973
0.3627	6800	0.0027	0.0024	0.9971
0.368	6900	0.0025	0.0025	0.9971
0.3733	7000	0.0034	0.0023	0.9976
0.3787	7100	0.0027	0.0023	0.9975
0.384	7200	0.0029	0.0025	0.9974
0.3893	7300	0.003	0.0024	0.9976
0.3947	7400	0.0022	0.0022	0.9975
0.4	7500	0.0023	0.0022	0.9974
0.4053	7600	0.0028	0.0022	0.9973
0.4107	7700	0.0027	0.0022	0.9974
0.416	7800	0.004	0.0024	0.9972
0.4213	7900	0.0027	0.0024	0.9977
0.4267	8000	0.0043	0.0023	0.9977
0.432	8100	0.0031	0.0021	0.9977
0.4373	8200	0.0027	0.0022	0.9974
0.4427	8300	0.0031	0.0022	0.9973
0.448	8400	0.0023	0.0022	0.9975
0.4533	8500	0.0028	0.0022	0.9973
0.4587	8600	0.0027	0.0021	0.9976
0.464	8700	0.0027	0.0021	0.9977
0.4693	8800	0.0024	0.0021	0.9977
0.4747	8900	0.0027	0.0021	0.9979
0.48	9000	0.0025	0.0022	0.9974
0.4853	9100	0.0027	0.0022	0.9974
0.4907	9200	0.0035	0.0019	0.9978
0.496	9300	0.0027	0.0019	0.9978
0.5013	9400	0.0028	0.0019	0.9979
0.5067	9500	0.0023	0.0019	0.9979
0.512	9600	0.0015	0.0019	0.9979
0.5173	9700	0.0023	0.0019	0.9979
0.5227	9800	0.0021	0.0018	0.9979
0.528	9900	0.0019	0.0018	0.9979
0.5333	10000	0.0029	0.0018	0.9981
0.5387	10100	0.003	0.0019	0.9978
0.544	10200	0.0021	0.0019	0.9978
0.5493	10300	0.0022	0.0019	0.9978
0.5547	10400	0.0018	0.0018	0.9978
0.56	10500	0.0017	0.0019	0.9977
0.5653	10600	0.0023	0.0018	0.9978
0.5707	10700	0.0017	0.0017	0.9979
0.576	10800	0.0023	0.0018	0.998
0.5813	10900	0.0025	0.0018	0.9978
0.5867	11000	0.0024	0.0017	0.9979
0.592	11100	0.0022	0.0018	0.9979
0.5973	11200	0.0028	0.0017	0.9980
0.6027	11300	0.0016	0.0017	0.9978
0.608	11400	0.0021	0.0017	0.9980
0.6133	11500	0.0013	0.0017	0.9981
0.6187	11600	0.0022	0.0017	0.9980
0.624	11700	0.0022	0.0019	0.9976
0.6293	11800	0.0017	0.0018	0.9977
0.6347	11900	0.0024	0.0017	0.9980
0.64	12000	0.0023	0.0017	0.9980
0.6453	12100	0.0019	0.0016	0.9981
0.6507	12200	0.0016	0.0016	0.9982
0.656	12300	0.0025	0.0016	0.9982
0.6613	12400	0.002	0.0016	0.9981
0.6667	12500	0.0019	0.0016	0.9981
0.672	12600	0.0021	0.0016	0.998
0.6773	12700	0.0023	0.0016	0.9980
0.6827	12800	0.0021	0.0016	0.9982
0.688	12900	0.0017	0.0016	0.9981
0.6933	13000	0.0026	0.0016	0.9981
0.6987	13100	0.0021	0.0016	0.9982
0.704	13200	0.0025	0.0016	0.9981
0.7093	13300	0.0019	0.0016	0.9980
0.7147	13400	0.0019	0.0016	0.9981
0.72	13500	0.0015	0.0016	0.9982
0.7253	13600	0.0023	0.0016	0.9980
0.7307	13700	0.0028	0.0016	0.9981
0.736	13800	0.0018	0.0015	0.9982
0.7413	13900	0.002	0.0016	0.9980
0.7467	14000	0.0023	0.0015	0.9983
0.752	14100	0.0017	0.0015	0.9981
0.7573	14200	0.0018	0.0015	0.9983
0.7627	14300	0.0023	0.0014	0.9983
0.768	14400	0.0014	0.0014	0.9982
0.7733	14500	0.0014	0.0014	0.9982
0.7787	14600	0.0012	0.0014	0.9982
0.784	14700	0.0022	0.0014	0.9982
0.7893	14800	0.0015	0.0014	0.9983
0.7947	14900	0.0018	0.0014	0.9983
0.8	15000	0.0019	0.0014	0.9984
0.8053	15100	0.0014	0.0014	0.9982
0.8107	15200	0.0017	0.0014	0.9982
0.816	15300	0.0014	0.0014	0.9983
0.8213	15400	0.0017	0.0014	0.9982
0.8267	15500	0.001	0.0014	0.9982
0.832	15600	0.0021	0.0014	0.9982
0.8373	15700	0.0013	0.0014	0.9982
0.8427	15800	0.002	0.0013	0.9983
0.848	15900	0.0014	0.0013	0.9983
0.8533	16000	0.0015	0.0013	0.9983
0.8587	16100	0.001	0.0013	0.9983
0.864	16200	0.0014	0.0013	0.9983
0.8693	16300	0.0013	0.0013	0.9983
0.8747	16400	0.0019	0.0013	0.9983
0.88	16500	0.0017	0.0013	0.9983
0.8853	16600	0.0014	0.0013	0.9984
0.8907	16700	0.0018	0.0013	0.9984
0.896	16800	0.001	0.0013	0.9984
0.9013	16900	0.0019	0.0013	0.9984
0.9067	17000	0.0022	0.0013	0.9985
0.912	17100	0.0017	0.0013	0.9985
0.9173	17200	0.002	0.0013	0.9986
0.9227	17300	0.0012	0.0013	0.9986
0.928	17400	0.0015	0.0013	0.9985
0.9333	17500	0.0012	0.0013	0.9985
0.9387	17600	0.0014	0.0013	0.9985
0.944	17700	0.0019	0.0013	0.9985
0.9493	17800	0.0018	0.0013	0.9985
0.9547	17900	0.001	0.0013	0.9985
0.96	18000	0.0016	0.0013	0.9986
0.9653	18100	0.0011	0.0013	0.9986
0.9707	18200	0.0016	0.0013	0.9986
0.976	18300	0.0024	0.0013	0.9986
0.9813	18400	0.0026	0.0013	0.9986
0.9867	18500	0.001	0.0013	0.9986
0.992	18600	0.0015	0.0013	0.9986
0.9973	18700	0.0017	0.0013	0.9986
1.0	18750	-	-	0.9987

Framework Versions

Python: 3.11.1
Sentence Transformers: 3.3.1
Transformers: 4.47.0
PyTorch: 2.1.1+cu121
Accelerate: 1.2.0
Datasets: 2.18.0
Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

bibtex

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

	anchor	positive	negative
type	string	string	string
details	min: 18 tokens mean: 61.91 tokens max: 247 tokens	min: 4 tokens mean: 11.04 tokens max: 39 tokens	min: 4 tokens mean: 11.29 tokens max: 50 tokens

anchor	positive	negative
	Supernumerary deciduous mandibular tooth	: {
	Hemorrhage into subdural space of neuraxis	+
	Deep mammalian bite wound	+

anchor	positive	negative
	Anastomosis of duodenum to colon	+
	Benign neoplasm of colon	+
	Closed traumatic dislocation of hip	+