ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 16.7 tokens max: 38 tokens	min: 26 tokens mean: 96.94 tokens max: 153 tokens

• Samples:

  anchor	positive
  What does EPIC agree about its FOIA request?	other documents which were made available to or prepared for or by” the Commission, a direct [object Object]quotation from section 10(b) of FACA. Pl.’s Mot. Exs. at 21. EPIC agrees that its FOIA request [object Object]“exactly track[s] the language of FACA § 10(b)”—i.e., that its FOIA request is meant to be [object Object]coterminous with FACA’s parameters. Pl.’s Mem. at 24; Pl.’s Reply at 9. [object Object]25
  What specific finding does Sussman, 494 F.3d at 1116 emphasize that the district court must make?	79 The Court need not assess the segregability efforts of the NSA because the plaintiff does not challenge any [object Object]withholding decisions made by the NSA, and thus the Court need not review any such withholding decisions. See, [object Object]e.g., Sussman, 494 F.3d at 1116 (holding that “the district court must make specific findings of segregability
  Who expressed a viewpoint on the application of the reasonable juror test?	test used by United States Courts of Appeals for authentication of social media evidence [object Object]because it was consistent with Maryland Rule 5-901. See id. at 366, 19 A.3d at 429 [object Object](Harrell, J., dissenting). Judge Harrell explained that, in his view, applying the reasonable [object Object]juror test would have led to the conclusion that the social media evidence at issue was

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• warmup_steps: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.2
• Transformers: 5.0.0
• PyTorch: 2.9.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss