This is a sentence-transformers model finetuned from tmnam20/ViPubMedT5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: tmnam20/ViPubMedT5
Maximum Sequence Length: 128 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: T5EncoderModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vimednli/vipubmedt5-w_multi-SynPD")
# Run inference
sentences = [
    'Việc cấy_ghép các EC và MC có nguồn_gốc từ các tế_bào ES chưa phân_biệt của con_người có khả_năng góp_phần tái_tạo mạch_máu điều_trị và do_đó làm giảm diện_tích nhồi máu sau đột_quỵ .',
    'Việc cấy_ghép các tế_bào mạch_máu có nguồn_gốc từ tế_bào gốc phôi người góp_phần tái_tạo mạch_máu sau đột_quỵ ở chuột .',
    'Những bệnh_nhân ET không bị mất trí_nhớ có nhiều thay_đổi liên_quan đến Alzheimer trong bệnh_lý thần_kinh tau hơn so với nhóm đối_chứng , cho thấy mối liên_hệ giữa bệnh_lý tau và run .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 92,842 training samples
Columns: sentence1 and sentence2
Approximate statistics based on the first 1000 samples:
sentence1 sentence2
type string string
details
min: 13 tokens
mean: 60.83 tokens
max: 128 tokens
min: 3 tokens
mean: 48.05 tokens
max: 128 tokens

Samples:

sentence1	sentence2
`Trong nghiên_cứu hiện_tại , hoạt_hoá P21 bằng hoạt_hoá gen do RNA ( RNAa ) gây ra hoạt_động chống khối_u trong ống_nghiệm trên dòng tế_bào u thần_kinh đệm SHG-44 của người .`	`dsRNA nhắm vào vùng khởi_động p21 ( dsP 21 ) đã gây cảm_ứng đáng_kể sự biểu_hiện của p21 ở mức phiên mã và protein , và làm giảm sự biểu_hiện của survivin .`
`Kết_quả của nghiên_cứu này cho thấy một sự tương_đồng về trình_tự không mong_đợi của các protein họ GH97 với glycoside hydrolase từ một_số họ khác , có cấu_trúc nếp gấp ( beta / alpha ) 8 của miền xúc_tác và cơ_chế giữ lại quá_trình thuỷ_phân liên_kết glycoside .`	`GH97 là một họ mới của glycoside hydrolase , có liên_quan đến họ siêu alpha-galactosidase.`
`MRI là một công_cụ hiệu_quả để dự_đoán đáp_ứng với NAC .`	`Phân giai MRI sau hoá_trị tân_bổ_trợ cho ung_thư vú : sinh_học khối_u có ảnh_hưởng đến độ_chính_xác không ?`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
per_device_eval_batch_size: 32
learning_rate: 3e-05
num_train_epochs: 10
warmup_ratio: 0.1

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 3e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 10
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
tp_size: 0
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0345	100	4.4987
0.0689	200	3.5509
0.1034	300	2.2814
0.1378	400	1.3726
0.1723	500	1.0296
0.2068	600	0.7233
0.2412	700	0.5698
0.2757	800	0.4624
0.3101	900	0.4061
0.3446	1000	0.3497
0.3790	1100	0.2957
0.4135	1200	0.2716
0.4480	1300	0.2456
0.4824	1400	0.2431
0.5169	1500	0.1974
0.5513	1600	0.2078
0.5858	1700	0.2016
0.6203	1800	0.2097
0.6547	1900	0.1855
0.6892	2000	0.1789
0.7236	2100	0.1753
0.7581	2200	0.1629
0.7926	2300	0.1748
0.8270	2400	0.1578
0.8615	2500	0.1452
0.8959	2600	0.1377
0.9304	2700	0.1379
0.9649	2800	0.1439
0.9993	2900	0.1434
1.0338	3000	0.1139
1.0682	3100	0.0966
1.1027	3200	0.1121
1.1371	3300	0.0996
1.1716	3400	0.1095
1.2061	3500	0.1031
1.2405	3600	0.1149
1.2750	3700	0.1239
1.3094	3800	0.0969
1.3439	3900	0.1044
1.3784	4000	0.1141
1.4128	4100	0.0894
1.4473	4200	0.1172
1.4817	4300	0.1009
1.5162	4400	0.0904
1.5507	4500	0.1198
1.5851	4600	0.0956
1.6196	4700	0.1061
1.6540	4800	0.0867
1.6885	4900	0.0908
1.7229	5000	0.1025
1.7574	5100	0.1099
1.7919	5200	0.0932
1.8263	5300	0.0848
1.8608	5400	0.1027
1.8952	5500	0.0851
1.9297	5600	0.0917
1.9642	5700	0.0883
1.9986	5800	0.0931
2.0331	5900	0.0625
2.0675	6000	0.0503
2.1020	6100	0.0627
2.1365	6200	0.0725
2.1709	6300	0.0529
2.2054	6400	0.0591
2.2398	6500	0.0501
2.2743	6600	0.0608
2.3088	6700	0.0616
2.3432	6800	0.0492
2.3777	6900	0.0556
2.4121	7000	0.0744
2.4466	7100	0.0661
2.4810	7200	0.0554
2.5155	7300	0.0615
2.5500	7400	0.0565
2.5844	7500	0.0628
2.6189	7600	0.0527
2.6533	7700	0.069
2.6878	7800	0.0666
2.7223	7900	0.0642
2.7567	8000	0.0601
2.7912	8100	0.0564
2.8256	8200	0.0549
2.8601	8300	0.0552
2.8946	8400	0.0692
2.9290	8500	0.0607
2.9635	8600	0.0537
2.9979	8700	0.0534
3.0324	8800	0.0365
3.0669	8900	0.041
3.1013	9000	0.0405
3.1358	9100	0.0362
3.1702	9200	0.0365
3.2047	9300	0.0451
3.2391	9400	0.0363
3.2736	9500	0.0444
3.3081	9600	0.0349
3.3425	9700	0.0445
3.3770	9800	0.0491
3.4114	9900	0.0429
3.4459	10000	0.0399
3.4804	10100	0.0364
3.5148	10200	0.0429
3.5493	10300	0.0394
3.5837	10400	0.0397
3.6182	10500	0.0406
3.6527	10600	0.038
3.6871	10700	0.0379
3.7216	10800	0.0392
3.7560	10900	0.0395
3.7905	11000	0.0331
3.8249	11100	0.0415
3.8594	11200	0.0421
3.8939	11300	0.0371
3.9283	11400	0.0333
3.9628	11500	0.0352
3.9972	11600	0.0371
4.0317	11700	0.0266
4.0662	11800	0.0288
4.1006	11900	0.0281
4.1351	12000	0.0318
4.1695	12100	0.0256
4.2040	12200	0.0275
4.2385	12300	0.0245
4.2729	12400	0.0295
4.3074	12500	0.0282
4.3418	12600	0.0286
4.3763	12700	0.0231
4.4108	12800	0.03
4.4452	12900	0.0244
4.4797	13000	0.0231
4.5141	13100	0.0222
4.5486	13200	0.027
4.5830	13300	0.0301
4.6175	13400	0.0256
4.6520	13500	0.0325
4.6864	13600	0.0291
4.7209	13700	0.0263
4.7553	13800	0.0215
4.7898	13900	0.0277
4.8243	14000	0.024
4.8587	14100	0.0242
4.8932	14200	0.0259
4.9276	14300	0.0279
4.9621	14400	0.0247
4.9966	14500	0.0285
5.0310	14600	0.0206
5.0655	14700	0.0183
5.0999	14800	0.0161
5.1344	14900	0.019
5.1688	15000	0.0198
5.2033	15100	0.0174
5.2378	15200	0.0157
5.2722	15300	0.0191
5.3067	15400	0.0181
5.3411	15500	0.0165
5.3756	15600	0.018
5.4101	15700	0.0194
5.4445	15800	0.0221
5.4790	15900	0.017
5.5134	16000	0.019
5.5479	16100	0.0166
5.5824	16200	0.0156
5.6168	16300	0.0248
5.6513	16400	0.0189
5.6857	16500	0.0188
5.7202	16600	0.0191
5.7547	16700	0.02
5.7891	16800	0.0157
5.8236	16900	0.0247
5.8580	17000	0.0218
5.8925	17100	0.0191
5.9269	17200	0.0141
5.9614	17300	0.0203
5.9959	17400	0.0169
6.0303	17500	0.0122
6.0648	17600	0.0128
6.0992	17700	0.0151
6.1337	17800	0.0162
6.1682	17900	0.0137
6.2026	18000	0.0124
6.2371	18100	0.0127
6.2715	18200	0.0152
6.3060	18300	0.0151
6.3405	18400	0.0164
6.3749	18500	0.0131
6.4094	18600	0.0155
6.4438	18700	0.0166
6.4783	18800	0.0149
6.5127	18900	0.0165
6.5472	19000	0.0181
6.5817	19100	0.014
6.6161	19200	0.0158
6.6506	19300	0.0171
6.6850	19400	0.0156
6.7195	19500	0.0143
6.7540	19600	0.0142
6.7884	19700	0.0151
6.8229	19800	0.0153
6.8573	19900	0.0141
6.8918	20000	0.0169
6.9263	20100	0.016
6.9607	20200	0.0128
6.9952	20300	0.0145
7.0296	20400	0.0103
7.0641	20500	0.0128
7.0986	20600	0.0088
7.1330	20700	0.0146
7.1675	20800	0.0101
7.2019	20900	0.0145
7.2364	21000	0.0141
7.2708	21100	0.0098
7.3053	21200	0.011
7.3398	21300	0.0117
7.3742	21400	0.0115
7.4087	21500	0.0129
7.4431	21600	0.0121
7.4776	21700	0.0096
7.5121	21800	0.0125
7.5465	21900	0.0115
7.5810	22000	0.0147
7.6154	22100	0.0149
7.6499	22200	0.0133
7.6844	22300	0.0127
7.7188	22400	0.0137
7.7533	22500	0.0113
7.7877	22600	0.0136
7.8222	22700	0.0128
7.8567	22800	0.0127
7.8911	22900	0.0154
7.9256	23000	0.0118
7.9600	23100	0.0118
7.9945	23200	0.0128
8.0289	23300	0.0098
8.0634	23400	0.0103
8.0979	23500	0.0125
8.1323	23600	0.0109
8.1668	23700	0.0083
8.2012	23800	0.0112
8.2357	23900	0.0108
8.2702	24000	0.0113
8.3046	24100	0.0107
8.3391	24200	0.011
8.3735	24300	0.01
8.4080	24400	0.0104
8.4425	24500	0.0099
8.4769	24600	0.0106
8.5114	24700	0.0111
8.5458	24800	0.0111
8.5803	24900	0.0105
8.6147	25000	0.0091
8.6492	25100	0.0128
8.6837	25200	0.0125
8.7181	25300	0.0115
8.7526	25400	0.0119
8.7870	25500	0.0115
8.8215	25600	0.0073
8.8560	25700	0.0107
8.8904	25800	0.012
8.9249	25900	0.0113
8.9593	26000	0.0104
8.9938	26100	0.0124
9.0283	26200	0.0092
9.0627	26300	0.0129
9.0972	26400	0.0094
9.1316	26500	0.0109
9.1661	26600	0.0094
9.2006	26700	0.0098
9.2350	26800	0.0103
9.2695	26900	0.0097
9.3039	27000	0.0106
9.3384	27100	0.0079
9.3728	27200	0.0082
9.4073	27300	0.0095
9.4418	27400	0.0086
9.4762	27500	0.009
9.5107	27600	0.0089
9.5451	27700	0.0102
9.5796	27800	0.0111
9.6141	27900	0.0104
9.6485	28000	0.011
9.6830	28100	0.0096
9.7174	28200	0.0096
9.7519	28300	0.0106
9.7864	28400	0.0076
9.8208	28500	0.0079
9.8553	28600	0.0097
9.8897	28700	0.0083
9.9242	28800	0.0077
9.9586	28900	0.0104
9.9931	29000	0.0107

Framework Versions

Python: 3.9.19
Sentence Transformers: 4.1.0
Transformers: 4.51.3
PyTorch: 2.2.0+cu121
Accelerate: 0.34.2
Datasets: 2.19.1
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	sentence1	sentence2
type	string	string
details	min: 13 tokens mean: 60.83 tokens max: 128 tokens	min: 3 tokens mean: 48.05 tokens max: 128 tokens