ModernBERT Embed base Legal Matryoshka

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,692 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 3 tokens mean: 176.61 tokens max: 742 tokens	min: 12 tokens mean: 23.56 tokens max: 51 tokens

• Samples:

  positive	anchor
  # Section 3:Subsoil drainage3.1 The provisions which follow assume that the site of the building is not subject to general flooding (see paragraph 0.8 ) or, if it is, that appropriate steps are being taken.3.2 Where the water table can rise to within 0.25 m of the lowest floor of the building, or where surface water could enter or adversely affect the building, either the ground to be covered by the building should be drained by gravity, or other effective means of safeguarding the building should be taken.3.3 If an active subsoil drain is cut during excavation and if it passes under the building it should be:a. re-laid in pipes with sealed joints and have access points outside the building; orb. re-routed around the building; orc. re-run to another outfall (see Diagram 3).3.4 Where there is a risk that groundwater beneath or around the building could adversely affect the stability and properties of the ground, consideration should be given to site drainage or other protection (see Sec...	What measures should be taken if a building site has a water table that can rise to within 0.25 meters of the lowest floor?
  Easily accessibleEither:- a window or doorway, any part of which is within 2 m vertically of an accessible level surface such as the ground or basement level, or an access balcony, or- a window within 2 m vertically of a flat or sloping roof (with a pitch of less than $30^{\circ}$ ) that is within 3.5 m of ground level.Coupled assemblyA doorset and window that are supplied as separate self-contained frames and fixed together on site.	What criteria determine if a window or doorway is considered easily accessible in a building?
  Fuels such as bituminous coal, untreated wood or compressed paper are not smokeless or low-volatiles fuels.3.These appliances are known as 'exempted fireplaces'.2.7 For fireplaces with openings larger than $500 \mathrm{~mm} \times 550 \mathrm{~mm}$ or fireplaces exposed on two or more sides (such as a fireplace under a canopy or open on both sides of a central chimney breast) a way of showing compliance would be to provide a flue with a cross-sectional area equal to 15 per cent of the total face area of the fireplace opening(s) (see Appendix B).However, specialist advice should be sought when proposing to construct flues having an area of:a. more than 15 per cent of the total face area of the fireplace openings; or	What is the required flue cross-sectional area for fireplaces with openings larger than 500 mm x 550 mm or exposed on multiple sides?

• Loss: MatryoshkaLoss with these parameters:
  json

  1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 10
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.2
• Sentence Transformers: 3.4.1
• Transformers: 4.50.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

MatryoshkaLoss

MultipleNegativesRankingLoss