ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,580 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 15.23 tokens max: 31 tokens	min: 25 tokens mean: 664.41 tokens max: 2429 tokens

• Samples:

  anchor	positive
  When was the obligation for the controller to respond to requests from the data subject set to take effect?	Modalities should be provided for facilitating the exercise of the data subject's rights under this Regulation, including mechanisms to request and, if applicable, obtain, free of charge, in particular, access to and rectification or erasure of personal data and the exercise of the right to object. The controller should also provide means for requests to be made electronically, especially where personal data are processed by electronic means. The controller should be obliged to respond to requests from the data subject without undue delay and at the latest within one month and to give reasons where the controller does not intend to comply with any such requests. 4.5.2016 L 119/11 Official Journal of the European Union EN
  Where can disputes be mediated or resolved?	The term technological measures means any technology, device or component designed to prevent or restrict unauthorized acts in respect of works or other subject-matter. Effective measures include access control or protection processes like encryption or copy control mechanisms.[object Object]It is prohibited to circumvent effective technological measures without permission from the rightholder, when done knowingly or with reasonable grounds to know.[object Object]It is prohibited to manufacture, import, distribute, sell, rent, advertise, or possess for commercial purposes devices or services intended to circumvent technological measures.[object Object]Violation of the provisions is punishable by imprisonment of at least one year and a fine of 2,900 to 15,000 Euro. It also entails civil sanctions under article 65 of Law 2121/1993.[object Object]Rightholders must ensure exceptions for reproduction for private use, teaching, libraries, judicial purposes, or accessibility for people with disabilities. Disputes may be mediated or resolved by the...
  How much was the defendant ordered to pay?	Court (Civil/Criminal):[object Object]Provisions:[object Object]Time of commission of the act:[object Object]Outcome (not guilty, guilty): ORDERS the defendant to pay the plaintiff the amount of two thousand four hundred thirty-four euros and eighty-three cents (€2,434.83) with legal interest from the service of the lawsuit.[object Object][object Object]Reasoning: Law 4537/2018 introduces mandatory provisions in favor of users, as according to Article 103, payment service providers are prohibited from deviating from the provisions to the detriment of payment service users, unless the possibility of deviation is expressly provided, and they can decide to offer only more favorable terms to payment service users. Under this law and its provisions, providers are only liable when there are unusual and unforeseen circumstances beyond the control of the party invoking them, and whose consequences could not have been avoided despite efforts to the contrary. However, operational risks and security risks of the system do not constitute unusual and unforeseen circu...

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 15
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.8.1
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss