SentenceTransformer based on answerdotai/ModernBERT-base

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 4096 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • code_search_net

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 4096, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Training Details

Training Dataset

code_search_net

• Dataset: code_search_net
• Size: 412,178 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 4 tokens mean: 73.72 tokens max: 2258 tokens	min: 46 tokens mean: 300.87 tokens max: 3119 tokens

• Samples:

  query	positive
  Extracts the list of arguments that start with any of the specified prefix values	def findArgs(args, prefixes):[object Object] """[object Object] Extracts the list of arguments that start with any of the specified prefix values[object Object] """[object Object] return list([[object Object] arg for arg in args[object Object] if len([p for p in prefixes if arg.lower().startswith(p.lower())]) > 0[object Object] ])
  Removes any arguments in the supplied list that are contained in the specified blacklist	def stripArgs(args, blacklist):[object Object] """[object Object] Removes any arguments in the supplied list that are contained in the specified blacklist[object Object] """[object Object] blacklist = [b.lower() for b in blacklist][object Object] return list([arg for arg in args if arg.lower() not in blacklist])
  Executes a child process and captures its output	def capture(command, input=None, cwd=None, shell=False, raiseOnError=False):[object Object] """[object Object] Executes a child process and captures its output[object Object] """[object Object] [object Object] # Attempt to execute the child process[object Object] proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, shell=shell, universal_newlines=True)[object Object] (stdout, stderr) = proc.communicate(input)[object Object] [object Object] # If the child process failed and we were asked to raise an exception, do so[object Object] if raiseOnError == True and proc.returncode != 0:[object Object] raise Exception([object Object] 'child process ' + str(command) +[object Object] ' failed with exit code ' + str(proc.returncode) +[object Object] '\nstdout: "' + stdout + '"' +[object Object] '\nstderr: "' + stderr + '"'[object Object] )[object Object] [object Object] return CommandOutput(proc.returncode, stdout, stderr)

• Loss: MultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Evaluation Dataset

code_search_net

• Dataset: code_search_net
• Size: 23,107 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 5 tokens mean: 168.27 tokens max: 2118 tokens	min: 48 tokens mean: 467.9 tokens max: 4096 tokens

• Samples:

  query	positive
  Train a deepq model.[object Object][object Object] Parameters[object Object] -------[object Object] env: gym.Env[object Object] environment to train on[object Object] network: string or a function[object Object] neural network to use as a q function approximator. If string, has to be one of the names of registered models in baselines.common.models[object Object] (mlp, cnn, conv_only). If a function, should take an observation tensor and return a latent variable tensor, which[object Object] will be mapped to the Q function heads (see build_q_func in baselines.deepq.models for details on that)[object Object] seed: int or None[object Object] prng seed. The runs with the same seed "should" give the same results. If None, no seeding is used.[object Object] lr: float[object Object] learning rate for adam optimizer[object Object] total_timesteps: int[object Object] number of env steps to optimizer for[object Object] buffer_size: int[object Object] size of the replay buffer[object Object] exploration_fraction: float[object Object] fraction of entire training period over which the exploration rate is annealed[object Object] exploration_final_eps: float[object Object] final value of ra...	def learn(env,[object Object] network,[object Object] seed=None,[object Object] lr=5e-4,[object Object] total_timesteps=100000,[object Object] buffer_size=50000,[object Object] exploration_fraction=0.1,[object Object] exploration_final_eps=0.02,[object Object] train_freq=1,[object Object] batch_size=32,[object Object] print_freq=100,[object Object] checkpoint_freq=10000,[object Object] checkpoint_path=None,[object Object] learning_starts=1000,[object Object] gamma=1.0,[object Object] target_network_update_freq=500,[object Object] prioritized_replay=False,[object Object] prioritized_replay_alpha=0.6,[object Object] prioritized_replay_beta0=0.4,[object Object] prioritized_replay_beta_iters=None,[object Object] prioritized_replay_eps=1e-6,[object Object] param_noise=False,[object Object] callback=None,[object Object] load_path=None,[object Object] **network_kwargs[object Object] ):[object Object] """Train a deepq model.[object Object][object Object] Parameters[object Object] -------[object Object] env: gym.Env[object Object] environment to train on[object Object] network: string or a function[object Object] neural network to use as a q function approximator. If string, has to be one of the ...
  Save model to a pickle located at [object Object]	def save_act(self, path=None):[object Object] """Save model to a pickle located at [object Object]"""[object Object] if path is None:[object Object] path = os.path.join(logger.get_dir(), "model.pkl")[object Object][object Object] with tempfile.TemporaryDirectory() as td:[object Object] save_variables(os.path.join(td, "model"))[object Object] arc_name = os.path.join(td, "packed.zip")[object Object] with zipfile.ZipFile(arc_name, 'w') as zipf:[object Object] for root, dirs, files in os.walk(td):[object Object] for fname in files:[object Object] file_path = os.path.join(root, fname)[object Object] if file_path != arc_name:[object Object] zipf.write(file_path, os.path.relpath(file_path, td))[object Object] with open(arc_name, "rb") as f:[object Object] model_data = f.read()[object Object] with open(path, "wb") as f:[object Object] cloudpickle.dump((model_data, self._act_params), f)
  CNN from Nature paper.	def nature_cnn(unscaled_images, **conv_kwargs):[object Object] """[object Object] CNN from Nature paper.[object Object] """[object Object] scaled_images = tf.cast(unscaled_images, tf.float32) / 255.[object Object] activ = tf.nn.relu[object Object] h = activ(conv(scaled_images, 'c1', nf=32, rf=8, stride=4, init_scale=np.sqrt(2),[object Object] **conv_kwargs))[object Object] h2 = activ(conv(h, 'c2', nf=64, rf=4, stride=2, init_scale=np.sqrt(2), **conv_kwargs))[object Object] h3 = activ(conv(h2, 'c3', nf=64, rf=3, stride=1, init_scale=np.sqrt(2), **conv_kwargs))[object Object] h3 = conv_to_fc(h3)[object Object] return activ(fc(h3, 'fc1', nh=512, init_scale=np.sqrt(2)))

• Loss: MultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_steps: 1000
• fp16: True

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

ModernBERT

Sentence Transformers

MultipleNegativesRankingLoss