CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:
  json

  1{ 2 "at_k": 10, 3 "always_rerank_positives": true 4}

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:
  json

  1{ 2 "dataset_names": [ 3 "msmarco", 4 "nfcorpus", 5 "nq" 6 ], 7 "rerank_k": 100, 8 "at_k": 10, 9 "always_rerank_positives": true 10}

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 10 characters mean: 34.15 characters max: 100 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  is dna acidic	['Great question. Deoxyribonucleic Acid is made up of a sugar (the deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion is where the acidity is, its also called phosphoric acid while in your cells.', 'Confidence votes 134. Great question. Deoxyribonucleic Acid is made up of a sugar (the deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion is where the acidity is, its also called phosphoric acid while in your cells.', 'Yes, DNA is an acid. In the structure of DNA, there is sugar (deoxyribose), a phosphate ion, and a nitrogenous base. the phosphate ion gives the acidic property. Fri Jan 14, 2005 5:07 pm. Which is kind of funny because the nitrogen base really IS a base.', 'First of all, DNA is not made up of nucleotide bases but of nucleotides. These consist of a sugar bound to one of the 4 nucleobases Adenine, Cytosine, Guanine or Thymine (Uracil in the case of RNA) and a phosphate group.', 'As you already know, the letters in DNA stand for Deoxyr...	[1, 1, 0, 0, 0, ...]
  what is a nerve conduction study	['A nerve conduction study (NCS), also called a nerve conduction velocity (NCV) test--is a measurement of the speed of conduction of an electrical impulse through a nerve. NCS can determine nerve damage and destruction. During the test, the nerve is stimulated, usually with surface electrode patches attached to the skin. The nerve conduction velocity (speed) is then calculated by measuring the distance between electrodes and the time it takes for electrical impulses to travel between electrodes. A related procedure that may be performed is electromyography (EMG).', 'Nerve conduction studies and needle EMG are commonly performed by physical medicine and rehabilitation or neurology specialists to assess the ability of the nervous system to conduct electrical impulses and to evaluate nerve/muscle function to determine if neuromuscular disease is present. Motor nerve conduction studies. In motor nerve conduction studies, motor nerves are stimulated and the compound muscle action potential ...	[1, 0, 0, 0, 0, ...]
  average nhl salary	['Most NHL teams, 17 of them, paid an average salary per player ranging from $1.8 million to $1 million. Only five teams--Philadelphia Flyers, Colorado Avalanche, New York Islanders, Anaheim Ducks and the Tampa Bay Lightning paid average salaries of under a million dollars per player. The minimum NHL player salary, per the collective bargaining agreement, is $500,000 in 2009-10 and 2010-11. The maximum seasonal salary for players new to the NHL is $900,000 for 2009 and 2010; and $925,000 for 2011 draftees.', 'Sports NHL NHL Salaries 2015. National Hockey League salary cap is what every team has to spend a limited amount of money on purchasing players every year. So that makes it more compatible to earn good players in all the teams and make it more balanced in strength. This led some teams to trade away well paid star players to fit the cap. But the teams were allowed to spend $70.4 Million in a year for the prorated shorted season length. The highest NHL career salary earns as of now ...	[1, 0, 0, 0, 0, ...]

• Loss: ListMLELoss with these parameters:
  json

  1{ 2 "lambda_weight": "sentence_transformers.cross_encoder.losses.ListMLELoss.ListMLELambdaWeight", 3 "activation_fct": "torch.nn.modules.linear.Identity", 4 "mini_batch_size": 16, 5 "respect_input_order": true 6}

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 1000 samples:

  	query	docs	labels
  type	string	list	list
  details	min: 12 characters mean: 33.4 characters max: 93 characters	min: 3 elements mean: 6.50 elements max: 10 elements	min: 3 elements mean: 6.50 elements max: 10 elements

• Samples:

  query	docs	labels
  what is contamination	['Contamination is the presence of an unwanted constituent, contaminant or impurity in a material, physical body, natural environment, workplace, etc. Contamination may include residual radioactive material remaining at a site after the completion of decommissioning of a site where there was a nuclear reactor, such as a power plant, experimental reactor, isotope reactor or a nuclear powered ship or submarine.', 'contamination. 1. the soiling or making inferior by contact or mixture, as by introduction of infectious organisms into a wound, into water, milk, food or onto the external surface of the body or on bandages and other dressings. 2. the deposition of radioactive material in any place where it is not desired. See Cross contamination Public health The presence of any foreign or undesired material in a system–eg, toxic contamination of the ground water in an ecosystem or untreated sewage into a stream Radiation physics The deposition of radioactive material in any place where it is...	[1, 0, 0, 0, 0, ...]
  what does crisis representation in social research mean	['The crisis of representation that now seems so apparent after the writing of Baudrillard was also the result of a convergence of historical conditions both inside and outside of art. As a result, art’s capacity to depict the world was effected. Digging through all of my archived materials continues to be a good distraction from being productive. I stumbled upon this diatribe… but can’t figure out if I wrote this for a class or for my own geekful bliss…. Issues surrounding representation have played a key role in the development of postmodern art.', 'CRISIS OF REPRESENTATION. This phrase was coined by George Marcus and Michael Fischer to refer specifically to the uncertainty within the human sciences about adequate means of describing social reality. This crisis arises from the (noncontroversial) claim that no interpretive account can ever directly or completely capture lived experience. Broadly conceived, the crisis is part of a more general set of ideas across the human sciences tha...	[1, 1, 0, 0, 0, ...]
  what was moche	['The Moche civilization (alternatively, the Mochica culture, Early Chimu, Pre-Chimu, Proto-Chimu, etc.) flourished in northern Peru with its capital near present-day Moche and Trujillo, from about 100 AD to 800 AD, during the Regional Development Epoch.', 'Moche Politics and Economy. The Moche were a stratified society with a powerful elite and an elaborate, well-codified ritual process. The political economy was based on the presence of large civic-ceremonial centers that produced a wide range of goods which were marketed to rural agrarian villages.', 'The Moche civilization (also known as the Mochica) flourished along the northern coast and valleys of ancient Peru, in particular, in the Chicama and Trujillo Valleys, between 1 CE and 800 CE.', 'While Mochica has been used in place of Moche by people describing this culture, the word Mochica actually refers to a particular dialect. This dialect, however, was not proven to be the dialect of the Moche Civilization and culture.', 'Moche ...	[1, 0, 0, 0, 0, ...]

• Loss: ListMLELoss with these parameters:
  json

  1{ 2 "lambda_weight": "sentence_transformers.cross_encoder.losses.ListMLELoss.ListMLELambdaWeight", 3 "activation_fct": "torch.nn.modules.linear.Identity", 4 "mini_batch_size": 16, 5 "respect_input_order": true 6}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.4.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

ListMLELoss