This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: nomic-ai/modernbert-embed-base
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("schaitanya/modernbert-embed-base-transcript-documents-v1")
# Run inference
sentences = [
    'What is the recommendation for a water birth if a woman has COVID-19?',
    '- Specific medications and antibody treatment.\nIf you are very unwell, your healthcare team may advise that your baby needs to be born early to help with your own treatment and recovery. How and when this may happen will depend on your individual situation.\nIf I have COVID-19, will this affect where I give birth and my choice of pain relief in labour?\nIf you have symptoms and have tested positive for COVID-19 at the time of birth:\nIt is recommended that you give birth in a consultant led maternity unit where you and your baby can be monitored more closely during labour.\nIt is safe for you to have a vaginal birth, and if you and your baby are both well you do not need to have a planned caesarean birth. Your birth choices should be respected and followed as closely as possible.\nA caesarean birth may be recommended if you or your baby are unwell or there are other complications. However, your chance of needing an emergency caesarean birth may be higher than usual.\nAll the usual options for pain relief for labour and birth are available to you, however a water birth is not recommended. This is because it is harder to monitor and give you any treatments needed.\nIf I have COVID-19, will this affect care of my baby after birth?\nIf your baby is well and does not require care in the neonatal unit, you will stay together after you have given birth. Skin-to-skin contact is encouraged.\nHow you feed your baby is dependent on your own circumstances and preferences, and your choices will be supported. Breastfeeding may help pass protection from infections (including COVID-19) to your baby. There is no strong evidence to show that COVID-19 can be passed on in breast milk.',
    "(0:00 - 4:09)\nSo what is the color of the cake? Pink or blue? What is the color of the nursery? Pink or blue? And what about the baby's clothes? Are they frilly skirts or the soccer shirts? Have you ever wondered what is it which decides the sex of the baby inside? How that little pea-shaped embryo grows into that little baby girl or a baby boy? Since ages it was the mother who was held responsible for the sex of the baby. But now we know it is the father who decides whether it is going to be a pretty little daughter or a handsome baby boy. So hello everyone, this is Dr. Anjali Kumar once again bringing you greetings from Maitri.\n\nMaitri is a space where we talk anything and everything about women's health. So today we are starting our pregnancy series season 2 with this very question which every parent wants to know. So the baby inherits its genes from both the parents.\n\nThe genes are present in the DNA and the DNA is present in the chromosomes and the chromosomes are present in the nucleus of the cell. Every human cell has 23 pairs of chromosomes. So total 46 chromosomes.\n\nEach pair inherited from one parent. 22 of these pairs are called autosomes. They look the same in both males and females.\n\nThe 23rd pair, the sex chromosomes, that's the special one. It differs between males and females. Females have two copies of X chromosomes which makes it XX, while males have one X and one Y which makes it XY.\n\nSo at the time of fertilization, the father's sperm and the mother's egg each contributes one sex chromosomes to the baby. The mother can contribute only X since it has two copies of X chromosomes only, while the father can contribute either X or Y chromosomes. So the baby's biological or the genetic sex which is male or female is determined by the chromosome which the father contributes.\n\nSo if the father contributes his Y chromosome, it will be a male baby which is XY, while if he contributes the X chromosome, it will be a female baby XX. Baby's sex is determined at the time of fertilization or the conception when the sperm fertilizes the egg. Now this typically happens around day 14 to maybe day 17 in women who have regular cycles.\n\nNow this is the time when you don't even know that you are pregnant. You might not be even expecting a pregnancy. This is the time when the baby's sex is decided.\n\nAfter that, nobody and nothing can ever change the genetic sex of the baby. No medicine, no food, no kada, no jariputi, no exercise can change the sex of the baby afterwards.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5191
cosine_accuracy@3	0.7441
cosine_accuracy@5	0.8
cosine_accuracy@10	0.8912
cosine_precision@1	0.5191
cosine_precision@3	0.248
cosine_precision@5	0.16
cosine_precision@10	0.0891
cosine_recall@1	0.5191
cosine_recall@3	0.7441
cosine_recall@5	0.8
cosine_recall@10	0.8912
cosine_ndcg@10	0.7051
cosine_mrr@10	0.6457
cosine_map@100	0.6503

Training Details

Training Dataset

Unnamed Dataset

Size: 6,116 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 7 tokens
mean: 15.08 tokens
max: 33 tokens
min: 38 tokens
mean: 389.33 tokens
max: 683 tokens

Samples:

anchor	positive
`What lifestyle changes are recommended before pregnancy?`	(0:02 - 0:50)[object Object]Are you excited to be a father? How do you think you can help your wife or partner in this journey? Would you want to help your wife or partner during labour? Do you know how does the delivery occurs? Have you read something about the delivery in the baby care? So a lot has been written about women and pregnancy, but we do not talk much about the fathers. Is their role only up to providing the sperm to fertilize the egg? Is it all about the moms? So hello everyone, this is Dr. Anjali Kumar, once again bringing greetings from Maitri. Maitri is a space where we talk anything and everything about women's health.[object Object][object Object](0:50 - 1:06)[object Object]But this time in this episode, we will talk about the fathers. We were not sure when to plan our family. She wanted a baby early and I wanted to wait for a few years.[object Object][object Object](1:07 - 1:28)[object Object]Plan and talk when you want to plan the pregnancy. Plan well the career, the finances, visit a doctor together for the pre-conceptional checks, tests and the contraceptive ...
`Does the absence of symptoms indicate an absence of infection?`	`(0:00 - 0:21)[object Object]Very important point. Some people with STDs may not actually have any symptoms. Now this means that the person is a carrier of infection but she is absolutely capable of transmitting the infection to the other person.[object Object][object Object]So remember absence of symptoms does not mean absence of infection.`
`When does pre-eclampsia usually occur during pregnancy?`	What is pre-eclampsia?[object Object]Pre-eclampsia is a condition that usually happens after 20 weeks of pregnancy. The exact cause of pre-eclampsia is not understood. It is usually a combination of:[object Object]raised blood pressure (hypertension)[object Object]protein in your urine (proteinuria).[object Object]Sometimes pre-eclampsia can affect your liver, kidneys and blood clotting without protein in your urine.[object Object]Pre-eclampsia is common, affecting between 1–5 in 100 women during pregnancy. It is usually mild but in a small number of cases, it can develop into a more serious illness. Around one in 200 women develop severe pre-eclampsia, which can be life-threatening for both you and your baby.[object Object]How will I know if I have pre-eclampsia?[object Object]Often you will have no symptoms and pre-eclampsia may be diagnosed for the first time at your routine antenatal appointments or during labour when you have your blood pressure checked[object Object]and your urine tested.[object Object]If you do develop symptoms they usually happen towards the end of your pregnancy but can also happen f...

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 16
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
log_level: debug
bf16: True
tf32: True
load_best_model_at_end: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: debug
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	cosine_ndcg@10
0.4178	10	5.9152	-
0.8355	20	2.7824	-
0.9608	23	-	0.6781
1.2924	30	1.9575	-
1.7102	40	1.5202	-
1.9608	46	-	0.6943
2.1671	50	1.4008	-
2.5849	60	1.1741	-
2.9608	69	-	0.7031
3.0418	70	1.0995	-
3.4595	80	1.0416	-
3.8773	90	1.1648	-
3.9608	92	-	0.7051

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.11.11
Sentence Transformers: 3.4.1
Transformers: 4.49.0
PyTorch: 2.6.0+cu124
Accelerate: 1.3.0
Datasets: 3.4.1
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	anchor	positive
type	string	string
details	min: 7 tokens mean: 15.08 tokens max: 33 tokens	min: 38 tokens mean: 389.33 tokens max: 683 tokens