This is a sentence-transformers model finetuned from NeuML/pubmedbert-base-embeddings. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: NeuML/pubmedbert-base-embeddings
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("manojkumaryalaga/medrag-x-pubmedbert-v3")
# Run inference
sentences = [
    'Can Good Infection Control Be Obtained in One-stage Exchange of the Infected TKA to a Rotating Hinge Design?',
    'Prosthetic joint infection (PJI) occurs in 1% to 2% of total knee arthroplasties (TKAs). Although two-stage exchange is the preferred management method of patients with chronic PJI in TKA in North America, one-stage exchange is an alternative treatment method, but long-term studies of this approach have not been conducted.QUESTIONS/ We reviewed our minimum 9-year results of 70 patients who underwent one-stage exchange arthroplasty with a rotating hinge design to determine: (1) What was the proportion of pat',
    'The study compares physicians and the nursing staff of a hospital in terms of their extra-role behavior. Matters of interest include the extent of Organizational Citizenship Behavior (OCB) shown on the one hand and on the other hand which conditions stimulate the OCB of both physicians and nurses, respectively. The comparison was conducted by applying a questionnaire on n = 70 physicians and n = 112 nurses in a nursing department of a municipal hospital.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6533, 0.1460],
#         [0.6533, 1.0000, 0.4712],
#         [0.1460, 0.4712, 1.0000]])

Evaluation

Metrics

Information Retrieval

Dataset: pubmed-eval-1k
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.945
cosine_accuracy@3	0.975
cosine_accuracy@5	0.983
cosine_accuracy@10	0.989
cosine_precision@1	0.945
cosine_precision@3	0.325
cosine_precision@5	0.1966
cosine_precision@10	0.0989
cosine_recall@1	0.945
cosine_recall@3	0.975
cosine_recall@5	0.983
cosine_recall@10	0.989
cosine_ndcg@10	0.9683
cosine_mrr@10	0.9615
cosine_map@100	0.9618

Training Details

Training Dataset

Unnamed Dataset

Size: 12,000 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1
type string string
details
min: 6 tokens
mean: 43.72 tokens
max: 137 tokens
min: 19 tokens
mean: 88.21 tokens
max: 152 tokens

Samples:

sentence_0	sentence_1
Socioeconomic inequalities in cardiovascular mortality are well documented. The aim here is to examine the relation between childhood and adulthood class as well as the role of unique intergenerational social mobility trajectories in such mortality. Data were obtained from Swedish registries. Childhood and adulthood information were from the 1960 and 1990 censuses. Men born 1945-59 (809,199) were followed-up for four cardiovascular mortality outcomes 1990-2002 (5533 deaths) by means of Cox regressions. Thre	`Health-related quality-of-life (HRQOL) measures are being used more frequently in the evaluation of the adult deformity patient. This is due in part to the validation of the deformity-specific Scolios Research Society-22 (SRS-22). Hence, relationships between HRQOL outcomes and traditional measures of success such as deformity correction, fusion healing, and complications are being established. To examine the pattern of HRQOL outcome responses after adult deformity surgery.`
`Endovascular treatment of ruptured abdominal aortic aneurysms: is now EVAR the first choice of treatment?`	`This study was designed to evaluate the effectiveness of endovascular treatment (EVAR) for ruptured abdominal aortic aneurysms (rAAAs). Between September 2005 and December 2012, 44 patients with rAAA suitable for endovascular repair underwent emergency EVAR. We did not consider hemodynamic instability to be a contraindication for EVAR.`
`Does loss of consciousness predict neuropsychological decrements after concussion?`	`To investigate the importance of loss of consciousness (LOC) in predicting neuropsychological test performance in a large sample of patients with head injury. Retrospective comparison of neuropsychological test results for patients who suffered traumatic LOC, no LOC, or uncertain LOC.`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false 5}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 32
per_device_eval_batch_size: 32
num_train_epochs: 5
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	pubmed-eval-1k_cosine_ndcg@10
1.0	188	-	0.9700
2.0	376	-	0.9708
2.6596	500	1.6363	-
3.0	564	-	0.9718
4.0	752	-	0.9683

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.2.3
Transformers: 5.0.0
PyTorch: 2.10.0+cu128
Accelerate: 1.12.0
Datasets: 4.8.3
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	sentence_0	sentence_1
type	string	string
details	min: 6 tokens mean: 43.72 tokens max: 137 tokens	min: 19 tokens mean: 88.21 tokens max: 152 tokens