ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 768 3}

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 512 3}

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 256 3}

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 128 3}

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "truncate_dim": 64 3}

Training Details

Training Dataset

json

• Dataset: json
• Size: 5,822 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 16.57 tokens max: 41 tokens	min: 15 tokens mean: 97.04 tokens max: 156 tokens

• Samples:

  anchor	positive
  Under what solicitations do all task orders qualify according to the Defendant?	orders. See SHS MJAR at 36–37; VCH MJAR at 36–37 (same). For the reasons discussed below, [object Object]this Court concludes that the correct interpretation of “feature” as used in Section 3306(c)(3) lies [object Object]between Plaintiffs’ and Defendant’s positions. [object Object]As Defendant argues all task orders contemplated under the Polaris Solicitations qualify as
  What type of project is related to the cost-reimbursement category?	2156–57, 2647–48. Further, offerors can earn additional points for Primary Relevant Experience [object Object]by submitting (1) projects completed for various government customers; (2) cost-reimbursement [object Object]12 [object Object] [object Object]projects; (3) task order awards on multiple-award contracts; (4) projects outside the contiguous [object Object]United States; (5) projects related to cybersecurity experience; and (6) projects demonstrating a
  Who drafted the one-sentence order that lacked stated reasons?	its discretion, a reviewing court looks to the trial court’s “stated justification for refusing to [object Object]modify” the order. Skolnick, 191 Ill. 2d at 226. [object Object]¶ 35 [object Object] [object Object]In the case at bar, the one-sentence April 25 order did not provide any reasons at all. The [object Object]losing party drafted the order without any stated reasons, although a lack of stated reasons may

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• tf32: False
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.7.0+cu128
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss