ModernBERT-base trained on MSMARCO with CachedMultipleNegativesRankingLoss

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 300 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (normalize): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Datasets: NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:
  json

  1{ 2 "dataset_names": [ 3 "msmarco", 4 "nfcorpus", 5 "nq" 6 ], 7 "dataset_id": "sentence-transformers/NanoBEIR-en" 8}

Training Details

Training Dataset

Unnamed Dataset

• Size: 496,910 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, and negative_4
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4
  type	string	string	string	string	string	string
  modality	text	text	text	text	text	text
  details	min: 11 characters mean: 34.59 characters max: 75 characters	min: 88 characters mean: 365.38 characters max: 867 characters	min: 132 characters mean: 332.38 characters max: 670 characters	min: 95 characters mean: 372.29 characters max: 983 characters	min: 86 characters mean: 364.95 characters max: 925 characters	min: 118 characters mean: 332.33 characters max: 752 characters

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4
  what are the liberal arts?	liberal arts. 1. the academic course of instruction at a college intended to provide general knowledge and comprising the arts, humanities, natural sciences, and social sciences, as opposed to professional or technical subjects.	Because Liberal Arts is a new and innovative programme, we have gathered the most commonly-asked questions from applicants and answered them here. If you have any questions about the Liberal Arts programme which aren't answered here, please contact undergraduate enquiries: Email: [object Object].	Why do we call the liberal arts liberal? The liberal in liberal arts, a cornerstone of the education of so many, has very little to do with political leanings; its roots can be traced to the Latin word liber, meaning “free, unrestricted.” Our language took the term from the Latin liberales artes, which described the education given to freeman and members of the upper classes, and involved ...	Associate in Liberal Arts Degree. Some subjects that are emphasized in a liberal arts associate's degree program include literature, sciences, history, foreign languages, mathematics and philosophy.	These are some of the key questions examined by the Arts disciplines. A liberal arts education is by nature broad and diverse, rather than narrow and specialized. Choosing courses from many disciplines gives you a wide and useful education. In the first year, an Arts student normally takes a variety of introductory courses.
  what is the mechanism of action of fibrinolytic or thrombolytic drugs?	Baillière's Clinical Haematology. 6 Mechanism of action of the thrombolytic agents. 6 Mechanism of action of the thrombolytic agents JEFFREY I. WEITZ Fibrin formed during the haemostatic, inflammatory or tissue repair process serves a temporary role, and must be degraded to restore normal tissue function and structure.	Fibrinogen is a soluble plasma protein that is converted to polymeric fibrin in response to damage to the vascular system. The clotting process is initiated when platelets aggregate at the wound site. Their disruption releases biologically active amines and a proteolytic cascade follows which culminates in the conversion of fibrinogen to fibrin.	Mechanism of Antiviral Action: Acyclovir is a synthetic purine nucleoside analogue with in vitro and in vivo inhibitory activity against herpes simplex virus types 1 (HSV-1), 2 (HSV-2), and varicella zoster virus (VZV).	Chlorpropamide. Ranitidine. Rifampin. Vancomycin. Drug-induced thrombocytopenia occurs typically when a drug bound to the platelet creates a new and “foreign” antigen, causing an immune reaction. This disorder is indistinguishable from ITP except for the history of drug ingestion.	Mechanism of action. Heparin binds to the enzyme inhibitor antithrombin III (AT), causing a conformational change that results in its activation through an increase in the flexibility of its reactive site loop. The activated AT then inactivates thrombin, factor Xa and other proteases.
  what is normal plat count	78 Followers. A. Platelets are the tiny blood cells that help stop bleeding by binding together to form a clump or plug at sites of injury inside blood vessels. A normal platelet count is between 150,000 and 450,000 platelets per microliter (one-millionth of a liter, abbreviated mcL).The average platelet count is 237,000 per mcL in men and 266,000 per mcL in women.8 Followers. A. Platelets are the tiny blood cells that help stop bleeding by binding together to form a clump or plug at sites of injury inside blood vessels. A normal platelet count is between 150,000 and 450,000 platelets per microliter (one-millionth of a liter, abbreviated mcL).	They are critical for the clotting of bl … ood and have a lifespan of about 10 days. The normal platelet count ranges from 150,000 - 400,000/ml and a reduction in platelets is referred to as thrombocytopenia.This may be mild (100,000 - 150,000/ml), moderate (50,000 - 100,000/mlƒw) or severe (less than 50,000/ml).he normal platelet count ranges from 150,000 - 400,000/ml and a reduction in platelets is referred to as thrombocytopenia. This may be mild (100,000 - 150,000/ml), moderate (50,000 - 100,000/mlƒw) or severe (less than 50,000/ml).	As with platelets, neutrophil levels may become depressed in patients on myelosuppressive therapy such as imatinib mesylate (also called IM, Gleevec or Glivec). The normal range of the percentage of neutrophils is between 45% and 70%.More important than the percentage of neutrophils is the absolute neutrophil count (ANC) , which should fall between 1.0 to 8.0 k/ul.s with platelets, neutrophil levels may become depressed in patients on myelosuppressive therapy such as imatinib mesylate (also called IM, Gleevec or Glivec). The normal range of the percentage of neutrophils is between 45% and 70%.	The white blood cell count is done by counting the number of white blood cells in a sample of blood. A normal WBC is in the range of 4,000 to 11,000 cells per microliter. A low WBC is called leukopenia. A high WBC is termed leukocytosis. A normal absolute neutrophil count (ANC) is in the range of 1,500 to 8,000 cells per microliter.	Another term for hematocrit is packed cell volume, or PCV. Values obtained in this test are dependent on the number and size of red blood cells. Normal hematocrit levels are 38.8% to 50% for adult men and 34.9% to 44.5% for adult women. Ranges for children 15 years old and younger will differ according to age and sex.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "mini_batch_size": 16, 5 "gather_across_devices": false, 6 "directions": [ 7 "query_to_doc" 8 ], 9 "partition_mode": "joint", 10 "hardness_mode": null, 11 "hardness_strength": 0.0 12}

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 1
• learning_rate: 6e-05
• warmup_steps: 0.1
• bf16: True
• per_device_eval_batch_size: 128
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Training Time

• Training: 59.5 minutes
• Evaluation: 1.4 minutes
• Total: 1.0 hours

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.5.0.dev0
• Transformers: 5.6.2
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0.dev0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

CachedMultipleNegativesRankingLoss