This is a sentence-transformers model finetuned from andorei/BERGAMOT-multilingual-GAT. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: andorei/BERGAMOT-multilingual-GAT
Maximum Sequence Length: 64 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'коронарных',
    'coronary',
    'C0018787',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.8483, -0.0581],
#         [ 0.8483,  1.0000, -0.0185],
#         [-0.0581, -0.0185,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 172,593 training samples
Columns: mention_text, concept_name, CUI, entity_type, hard_negative_1, hard_negative_2, hard_negative_3, hard_negative_4, hard_negative_5, hard_negative_6, hard_negative_7, hard_negative_8, hard_negative_9, hard_negative_10, hard_negative_11, and hard_negative_12

Approximate statistics based on the first 1000 samples:

	mention_text	concept_name	CUI	entity_type	hard_negative_1	hard_negative_2	hard_negative_3	hard_negative_4	hard_negative_5	hard_negative_6	hard_negative_7	hard_negative_8	hard_negative_9	hard_negative_10	hard_negative_11	hard_negative_12
type	string	string	string	string	string	string	string	string	string	string	string	string	string	string	string	string
details	min: 3 tokens mean: 6.66 tokens max: 12 tokens	min: 3 tokens mean: 8.89 tokens max: 25 tokens	min: 5 tokens mean: 5.95 tokens max: 7 tokens	min: 4 tokens mean: 4.21 tokens max: 5 tokens	min: 4 tokens mean: 8.96 tokens max: 19 tokens	min: 4 tokens mean: 10.43 tokens max: 34 tokens	min: 3 tokens mean: 9.46 tokens max: 23 tokens	min: 4 tokens mean: 9.48 tokens max: 24 tokens	min: 4 tokens mean: 8.7 tokens max: 22 tokens	min: 3 tokens mean: 10.47 tokens max: 22 tokens	min: 3 tokens mean: 8.82 tokens max: 30 tokens	min: 3 tokens mean: 10.43 tokens max: 19 tokens	min: 3 tokens mean: 9.64 tokens max: 23 tokens	min: 4 tokens mean: 9.81 tokens max: 23 tokens	min: 4 tokens mean: 9.15 tokens max: 18 tokens	min: 3 tokens mean: 8.93 tokens max: 18 tokens

Samples:

mention_text	concept_name	CUI	entity_type	hard_negative_1	hard_negative_2	hard_negative_3	hard_negative_4	hard_negative_5	hard_negative_6	hard_negative_7	hard_negative_8	hard_negative_9	hard_negative_10	hard_negative_11	hard_negative_12
`тяжелой острой респираторной инфекцией`	`severe acute respiratory infection`	`C3873497`	`DISO`	`upper respiratory infection acute`	`acute respiratory tract infection`	`acute respiratory disease`	`upper respiratory disease, acute`	`острая респираторная недостаточность`	`protracted upper respiratory infections`	`acute respiratory infection: [nos] or [pharyngotracheitis] or [tracheopharyngitis]`	`острый респираторный дистресс-синдром`	`острая дыхательная недостаточность`	`respiratory tract infection (& [acute lower] or [lower])`	`healthcare associated severe acute respiratory syndrome`	`острое диффузное инфильтративное заболевание легких`
`тяжелой острой респираторной инфекцией`	`severe acute respiratory infection (disorder)`	`C3873497`	`DISO`	`upper respiratory infection acute`	`acute respiratory tract infection`	`acute respiratory disease`	`upper respiratory disease, acute`	`острая респираторная недостаточность`	`protracted upper respiratory infections`	`acute respiratory infection: [nos] or [pharyngotracheitis] or [tracheopharyngitis]`	`острый респираторный дистресс-синдром`	`острая дыхательная недостаточность`	`respiratory tract infection (& [acute lower] or [lower])`	`healthcare associated severe acute respiratory syndrome`	`острое диффузное инфильтративное заболевание легких`
`тяжелой острой респираторной инфекцией`	`severe acute respiratory infection (sari)`	`C3873497`	`DISO`	`upper respiratory infection acute`	`acute respiratory tract infection`	`acute respiratory disease`	`upper respiratory disease, acute`	`острая респираторная недостаточность`	`protracted upper respiratory infections`	`acute respiratory infection: [nos] or [pharyngotracheitis] or [tracheopharyngitis]`	`острый респираторный дистресс-синдром`	`острая дыхательная недостаточность`	`respiratory tract infection (& [acute lower] or [lower])`	`healthcare associated severe acute respiratory syndrome`	`острое диффузное инфильтративное заболевание легких`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false, 5 "directions": [ 6 "query_to_doc" 7 ], 8 "partition_mode": "joint", 9 "hardness_mode": null, 10 "hardness_strength": 0.0 11}

Evaluation Dataset

Unnamed Dataset

Size: 22,650 evaluation samples
Columns: mention_text, concept_name, CUI, and entity_type

Approximate statistics based on the first 1000 samples:

	mention_text	concept_name	CUI	entity_type
type	string	string	string	string
details	min: 3 tokens mean: 6.07 tokens max: 12 tokens	min: 3 tokens mean: 7.46 tokens max: 19 tokens	min: 6 tokens mean: 6.0 tokens max: 6 tokens	min: 4 tokens mean: 4.25 tokens max: 5 tokens

Samples:

mention_text	concept_name	CUI	entity_type
`ишемической болезнью сердца`	`cardiac ischaemia`	`C0151744`	`DISO`
`ишемической болезнью сердца`	`cardiac ischemia`	`C0151744`	`DISO`
`ишемической болезнью сердца`	`disease, ischemic heart`	`C0151744`	`DISO`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false, 5 "directions": [ 6 "query_to_doc" 7 ], 8 "partition_mode": "joint", 9 "hardness_mode": null, 10 "hardness_strength": 0.0 11}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 64
num_train_epochs: 5.0
learning_rate: 2e-05
warmup_steps: 0.1
weight_decay: 0.01
per_device_eval_batch_size: 128
remove_unused_columns: False

All Hyperparameters

Click to expand

per_device_train_batch_size: 64
num_train_epochs: 5.0
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.1
optim: adamw_torch
optim_args: None
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 128
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: False
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss
0.0182	50	3.4020	-
0.0365	100	3.3501	-
0.0547	150	3.1245	-
0.0730	200	2.8373	-
0.0912	250	2.6900	-
0.1095	300	2.6216	-
0.1277	350	2.5536	-
0.1460	400	2.4949	-
0.1642	450	2.3477	-
0.1825	500	2.3037	-
0.2007	550	2.2350	-
0.2190	600	2.1277	-
0.2372	650	2.1636	-
0.2555	700	2.0572	-
0.2737	750	1.9219	-
0.2920	800	1.8304	-
0.3102	850	1.7910	-
0.3285	900	1.6376	-
0.3467	950	1.6142	-
0.3650	1000	1.5136	-
0.3832	1050	1.4201	-
0.4015	1100	1.4102	-
0.4197	1150	1.2804	-
0.4380	1200	1.2794	-
0.4562	1250	1.2063	-
0.4745	1300	1.0854	-
0.4927	1350	1.0697	-
0.5109	1400	1.0302	-
0.5292	1450	0.9603	-
0.5474	1500	0.8664	-
0.5657	1550	0.8549	-
0.5839	1600	0.7458	-
0.6022	1650	0.7424	-
0.6204	1700	0.6597	-
0.6387	1750	0.6438	-
0.6569	1800	0.5910	-
0.6752	1850	0.5584	-
0.6934	1900	0.5220	-
0.7117	1950	0.5056	-
0.7299	2000	0.4793	-
0.7482	2050	0.4925	-
0.7664	2100	0.4104	-
0.7847	2150	0.4002	-
0.8029	2200	0.3888	-
0.8212	2250	0.3699	-
0.8394	2300	0.3638	-
0.8577	2350	0.3388	-
0.8759	2400	0.4104	-
0.8942	2450	0.5003	-
0.9124	2500	0.6588	-
0.9307	2550	1.1744	-
0.9489	2600	1.7530	-
0.9672	2650	2.4516	-
0.9854	2700	3.0282	-
1.0	2740	-	1.3248
1.0036	2750	1.0572	-
1.0219	2800	1.5553	-
1.0401	2850	1.3146	-
1.0584	2900	1.1039	-
1.0766	2950	1.0263	-
1.0949	3000	0.8850	-
1.1131	3050	0.8426	-
1.1314	3100	0.8203	-
1.1496	3150	0.7419	-
1.1679	3200	0.7089	-
1.1861	3250	0.6440	-
1.2044	3300	0.5748	-
1.2226	3350	0.5992	-
1.2409	3400	0.5424	-
1.2591	3450	0.5291	-
1.2774	3500	0.4895	-
1.2956	3550	0.4538	-
1.3139	3600	0.4536	-
1.3321	3650	0.3768	-
1.3504	3700	0.3810	-
1.3686	3750	0.3734	-
1.3869	3800	0.3235	-
1.4051	3850	0.3409	-
1.4234	3900	0.3114	-
1.4416	3950	0.3189	-
1.4599	4000	0.2830	-
1.4781	4050	0.2600	-
1.4964	4100	0.2592	-
1.5146	4150	0.2503	-
1.5328	4200	0.2238	-
1.5511	4250	0.2194	-
1.5693	4300	0.2016	-
1.5876	4350	0.1909	-
1.6058	4400	0.1882	-
1.6241	4450	0.1657	-
1.6423	4500	0.1682	-
1.6606	4550	0.1644	-
1.6788	4600	0.1433	-
1.6971	4650	0.1404	-
1.7153	4700	0.1397	-
1.7336	4750	0.1471	-
1.7518	4800	0.1545	-
1.7701	4850	0.1426	-
1.7883	4900	0.1297	-
1.8066	4950	0.1487	-
1.8248	5000	0.1501	-
1.8431	5050	0.1655	-
1.8613	5100	0.1655	-
1.8796	5150	0.2310	-
1.8978	5200	0.3519	-
1.9161	5250	0.6200	-
1.9343	5300	1.1444	-
1.9526	5350	1.7849	-
1.9708	5400	2.4478	-
1.9891	5450	2.8559	-
2.0	5480	-	1.2179
2.0073	5500	0.6252	-
2.0255	5550	0.7470	-
2.0438	5600	0.6199	-
2.0620	5650	0.4969	-
2.0803	5700	0.4421	-
2.0985	5750	0.4175	-
2.1168	5800	0.3789	-
2.1350	5850	0.3827	-
2.1533	5900	0.3448	-
2.1715	5950	0.3282	-
2.1898	6000	0.2642	-
2.2080	6050	0.2654	-
2.2263	6100	0.2678	-
2.2445	6150	0.2470	-
2.2628	6200	0.2306	-
2.2810	6250	0.2398	-
2.2993	6300	0.1959	-
2.3175	6350	0.1915	-
2.3358	6400	0.1755	-
2.3540	6450	0.1971	-
2.3723	6500	0.1892	-
2.3905	6550	0.1430	-
2.4088	6600	0.1648	-
2.4270	6650	0.1512	-
2.4453	6700	0.1433	-
2.4635	6750	0.1331	-
2.4818	6800	0.1213	-
2.5	6850	0.1289	-
2.5182	6900	0.1186	-
2.5365	6950	0.1130	-
2.5547	7000	0.1096	-
2.5730	7050	0.1095	-
2.5912	7100	0.0938	-
2.6095	7150	0.0971	-
2.6277	7200	0.0909	-
2.6460	7250	0.0876	-
2.6642	7300	0.0803	-
2.6825	7350	0.0803	-
2.7007	7400	0.0828	-
2.7190	7450	0.0776	-
2.7372	7500	0.0958	-
2.7555	7550	0.0930	-
2.7737	7600	0.0938	-
2.7920	7650	0.0900	-
2.8102	7700	0.1081	-
2.8285	7750	0.1205	-
2.8467	7800	0.1308	-
2.8650	7850	0.1384	-
2.8832	7900	0.2105	-
2.9015	7950	0.3348	-
2.9197	8000	0.6486	-
2.9380	8050	1.2385	-
2.9562	8100	1.8995	-
2.9745	8150	2.5290	-
2.9927	8200	2.4490	-
3.0	8220	-	1.0663
3.0109	8250	0.3832	-
3.0292	8300	0.4348	-
3.0474	8350	0.3500	-
3.0657	8400	0.2919	-
3.0839	8450	0.2467	-
3.1022	8500	0.2613	-
3.1204	8550	0.2201	-
3.1387	8600	0.2125	-
3.1569	8650	0.2025	-
3.1752	8700	0.1824	-
3.1934	8750	0.1524	-
3.2117	8800	0.1455	-
3.2299	8850	0.1652	-
3.2482	8900	0.1462	-
3.2664	8950	0.1432	-
3.2847	9000	0.1474	-
3.3029	9050	0.1198	-
3.3212	9100	0.1083	-
3.3394	9150	0.1129	-
3.3577	9200	0.1341	-
3.3759	9250	0.0963	-
3.3942	9300	0.0921	-
3.4124	9350	0.1063	-
3.4307	9400	0.0951	-
3.4489	9450	0.0857	-
3.4672	9500	0.0813	-
3.4854	9550	0.0870	-
3.5036	9600	0.0808	-
3.5219	9650	0.0784	-
3.5401	9700	0.0763	-
3.5584	9750	0.0816	-
3.5766	9800	0.0692	-
3.5949	9850	0.0587	-
3.6131	9900	0.0666	-
3.6314	9950	0.0538	-
3.6496	10000	0.0693	-
3.6679	10050	0.0535	-
3.6861	10100	0.0506	-
3.7044	10150	0.0581	-
3.7226	10200	0.0595	-
3.7409	10250	0.0702	-
3.7591	10300	0.0668	-
3.7774	10350	0.0740	-
3.7956	10400	0.0665	-
3.8139	10450	0.0929	-
3.8321	10500	0.0941	-
3.8504	10550	0.1071	-
3.8686	10600	0.1266	-
3.8869	10650	0.2206	-
3.9051	10700	0.3430	-
3.9234	10750	0.7162	-
3.9416	10800	1.3359	-
3.9599	10850	2.0687	-
3.9781	10900	2.6235	-
3.9964	10950	1.9016	-
4.0	10960	-	0.9817
4.0146	11000	0.3683	-
4.0328	11050	0.3125	-
4.0511	11100	0.2377	-
4.0693	11150	0.2063	-
4.0876	11200	0.1838	-
4.1058	11250	0.1726	-
4.1241	11300	0.1587	-
4.1423	11350	0.1421	-
4.1606	11400	0.1515	-
4.1788	11450	0.1239	-
4.1971	11500	0.1038	-
4.2153	11550	0.1111	-
4.2336	11600	0.1105	-
4.2518	11650	0.1024	-
4.2701	11700	0.1059	-
4.2883	11750	0.1001	-
4.3066	11800	0.0848	-
4.3248	11850	0.0780	-
4.3431	11900	0.0821	-
4.3613	11950	0.0923	-
4.3796	12000	0.0727	-
4.3978	12050	0.0705	-
4.4161	12100	0.0721	-
4.4343	12150	0.0685	-
4.4526	12200	0.0589	-
4.4708	12250	0.0545	-
4.4891	12300	0.0755	-
4.5073	12350	0.0618	-
4.5255	12400	0.0609	-
4.5438	12450	0.0582	-
4.5620	12500	0.0566	-
4.5803	12550	0.0472	-
4.5985	12600	0.0519	-
4.6168	12650	0.0538	-
4.6350	12700	0.0413	-
4.6533	12750	0.0507	-
4.6715	12800	0.0432	-
4.6898	12850	0.0474	-
4.7080	12900	0.0423	-
4.7263	12950	0.0502	-
4.7445	13000	0.0609	-
4.7628	13050	0.0564	-
4.7810	13100	0.0623	-
4.7993	13150	0.0690	-
4.8175	13200	0.0782	-
4.8358	13250	0.0842	-
4.8540	13300	0.1017	-
4.8723	13350	0.1331	-
4.8905	13400	0.2390	-
4.9088	13450	0.3740	-
4.9270	13500	0.8491	-
4.9453	13550	1.5268	-
4.9635	13600	2.2905	-
4.9818	13650	2.9206	-
5.0	13700	1.2450	-

Training Time

Training: 2.6 hours
Evaluation: 45.4 seconds
Total: 2.6 hours

Framework Versions

Python: 3.10.12
Sentence Transformers: 5.4.1
Transformers: 5.6.2
PyTorch: 2.6.0+cu124
Accelerate: 1.13.0
Datasets: 4.8.4
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

bikingSolo/ru-dense-finetuning-infonce-12-hard-negatives-skip-7

SentenceTransformer based on andorei/BERGAMOT-multilingual-GAT

Model Details

Model Description

Model Sources

Full Model Architecture

Usage

Direct Usage (Sentence Transformers)

Training Details

Training Dataset

Unnamed Dataset

Evaluation Dataset

Unnamed Dataset

Training Hyperparameters

Non-Default Hyperparameters

All Hyperparameters

Training Logs

Training Time

Framework Versions

Citation

BibTeX

Sentence Transformers

MultipleNegativesRankingLoss

bikingSolo/
ru-dense-finetuning-infonce-12-hard-negatives-skip-7