ModernBERT Embed base fitness health Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 11,518 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 7 tokens mean: 239.56 tokens max: 410 tokens	min: 5 tokens mean: 10.8 tokens max: 26 tokens

• Samples:

  positive	anchor
  values and preferences among older people [object Object]in relation to exercise, noting that older [object Object]people valued the outcomes of exercise [object Object]for maintaining health. They judged that [object Object]the evidence for older people was likely to be relevant to all adults and agreed [object Object]there was likely to be some uncertainty or [object Object]variability with respect to people’s values and [object Object]preferences for exercise and its outcomes. [object Object]Some GDG members suggested that given reasonably consistent benefit and very [object Object]little harms, there would be no important [object Object]uncertainty or variability regarding people’s [object Object]values on the outcomes of exercise. In the [object Object]absence of direct qualitative evidence, the GDG judged from their own experience [object Object]that resource requirements for structured [object Object]exercise programmes would vary by country [object Object]and setting, but in some settings might [object Object]be associated with moderate costs (for [object Object]structured exercise programmes, compared with self-managed physical activity). The GDG [object Object]noted that costs could also vary according to [object Object]the modality of ...	exercise preferences and outcomes variability among adults
  ICRC, ICRC Hospital Design and Rehabilitation Guidelines, Vol. 1: Models Of Care, ICRC, Geneva, 2022: [object Object]. icrc.org/icrc-hospital-design-and-rehabilitation-guidelines-volume-1-models-of-care-print-en.html	ICRC rehabilitation guidelines 2022
  fitness training is guided by a health worker or (if feasible) performed self-directed by [object Object]the patient following education and advice.[object Object]Metacognitive [object Object]training[object Object]Metacognitive training aims to improve social functioning through reducing cognitive biases/psychotic symptoms (e.g. delusion, impaired self-awareness or insight). [object Object]Metacognitive training is usually provided as a structured group intervention during which participants perform exercises to reflect their own thinking and receive training in [object Object]strategies to cope with cognitive biases during daily routines. Metacognitive training is [object Object]guided by a health worker.[object Object]Mindfulness-[object Object]based approaches Mindfulness-based interventions aim to achieve a state of mindfulness in which a [object Object]person becomes more aware of their physical, mental, and emotional condition in the [object Object]present moment, without becoming judgemental. Mindfulness-based interventions (e.g. mindfulness-based cognitive therapy, acceptance and commitment therapy) [object Object]help people to pay attentio...	structured group interventions for metacognitive training

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.0.2
• Transformers: 4.51.1
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss