ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 26 tokens mean: 96.76 tokens max: 156 tokens	min: 8 tokens mean: 16.59 tokens max: 49 tokens

• Samples:

  positive	anchor
  properly authenticated. See id. at 367, 19 A.3d at 429 (Harrell, J., dissenting). [object Object]Four years later, in Sublet, 442 Md. at 637-38, 113 A.3d at 697-98, we adopted the [object Object]reasonable juror test for social media evidence and applied it in the three cases that were [object Object]consolidated for purposes of the opinion: Sublet v. State, Harris v. State, and Monge-	How many years after the dissent did the adoption of the reasonable juror test occur?
  to (1) a public-interest fee waiver, (2) the expedited processing of a request, or (3) the release of [object Object]information that implicates personal privacy, all are personal to a requester and thus cannot be [object Object]assigned. See, e.g., RTC Commercial Loan Trust 1995-NP1A v. Winthrop Mgmt., 923 F. Supp. [object Object]83, 88 (E.D. Va. 1996) (holding that “certain rights are purely personal and cannot be assigned”).	What type of fee waiver is mentioned as being personal to a requester?
  ‘IRO’] staff that reviews Agency records and makes public release determinations with an eye [object Object]toward evaluating directorate-specific equities.” Id. ¶ 4. Ms. Meeks also explains that “records [object Object]frequently involve the equities of multiple directorates,” and “[w]hen records implicate the [object Object]operational interests of multiple directorates, the reviews are conducted by the relevant IROs	Who conducts the reviews when the records implicate the operational interests of multiple directorates?

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.0.1
• Transformers: 4.50.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss