ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 29 tokens mean: 97.85 tokens max: 170 tokens	min: 7 tokens mean: 16.58 tokens max: 38 tokens

• Samples:

  positive	anchor
  n.2. But the Court cannot simply adopt this concession. “[S]ubject-matter jurisdiction, because [object Object]it involves a court’s power to hear a case, can never be forfeited or waived.” United States v. [object Object]Cotton, 535 U.S. 625, 630 (2002). The Court thus has “an independent obligation to determine [object Object]whether subject-matter jurisdiction exists.” Arbaugh v. Y&H Corp., 546 U.S. 500, 514 (2006).	According to Cotton, what can never be forfeited or waived?
  another because they involve common factual and legal issues. See Notice of Related Case, No. 11-444, ECF No. 2; [object Object]Notice of Related Case, No. 11-445, ECF No. 2. Although the Court has not formally consolidated these actions, [object Object]due to their interrelated nature and in the interests of judicial economy the Court has adjudicated dispositive motions	Has the Court formally consolidated the actions from the Notices of Related Case?
  [PROSECUTOR]: He’s authenticated it as to be the date and the time of the[object Object] [object Object]incident, it was a true and accurate reflection of that date and time. [object Object] [object Object]THE COURT: There are other questions you need to ask him, like, has he[object Object] [object Object]watched it. [object Object] [object Object][PROSECUTOR]: Okay. [object Object] [object Object]THE COURT: And is it a fair and accurate representation of what happened. [object Object]I mean, I’m not trying -- [object Object] [object Object][PROSECUTOR]: Okay.	What did the prosecutor confirm about the date and time?

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss