ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 28 tokens mean: 96.98 tokens max: 157 tokens	min: 8 tokens mean: 16.79 tokens max: 41 tokens

• Samples:

  positive	anchor
  After the bench conference concluded, the following exchange occurred between [object Object]the prosecutor and Mr. Zimmerman: [object Object][PROSECUTOR:] Did you watch this video in preparation? [object Object] [object Object][MR. ZIMMERMAN:] Yes, I did. [object Object] [object Object][PROSECUTOR:] Okay. And after seeing that video[,] was that a true and [object Object]accurate depiction of the events that occurred that day? [object Object] [object Object][MR. ZIMMERMAN:] Yes.	What was Mr. Zimmerman's response when asked if he watched the video in preparation?
  those guidelines still left a significant amount of ambiguity about “precisely what records [were] [object Object]being requested.” Id. (internal quotation marks omitted). Notably, although the plaintiff limited [object Object]the date range and number of reports requested, the plaintiff’s request would still place an [object Object]unreasonable search burden for two primary reasons. First, the plaintiff’s guideline asking for	What aspect of the plaintiff's request is mentioned as limited?
  motion without prejudice and permit him to do the same. See Prop. of the People, Inc., 330 [object Object]F. Supp. 3d at 390 (denying the parties’ motions without prejudice because the agency failed to [object Object]submit sufficient information justifying its FOIA withholdings and permitting both parties to file [object Object]renewed motions). [object Object]Thus, it is hereby ORDERED that Defendant’s Motion for Summary Judgment, ECF	What were the parties allowed to do after their motions were denied?

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.7.0+cu126
• Accelerate: 1.6.0
• Datasets: 3.5.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss