SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • code-search-net/code_search_net
• Language: code

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-dev
• Evaluated with EmbeddingSimilarityEvaluator

Training Details

Training Dataset

code-search-net/code_search_net

• Dataset: code-search-net/code_search_net
• Size: 20,000 training samples
• Columns: func_name and whole_func_string
• Approximate statistics based on the first 1000 samples:

  	func_name	whole_func_string
  type	string	string
  details	min: 3 tokens mean: 8.18 tokens max: 21 tokens	min: 38 tokens mean: 192.0 tokens max: 384 tokens

• Samples:

  func_name	whole_func_string
  ImageGraphCut.__msgc_step3_discontinuity_localization	def __msgc_step3_discontinuity_localization(self):[object Object] """[object Object] Estimate discontinuity in basis of low resolution image segmentation.[object Object] :return: discontinuity in low resolution[object Object] """[object Object] import scipy[object Object][object Object] start = self._start_time[object Object] seg = 1 - self.segmentation.astype(np.int8)[object Object] self.stats["low level object voxels"] = np.sum(seg)[object Object] self.stats["low level image voxels"] = np.prod(seg.shape)[object Object] # in seg is now stored low resolution segmentation[object Object] # back to normal parameters[object Object] # step 2: discontinuity localization[object Object] # self.segparams = sparams_hi[object Object] seg_border = scipy.ndimage.filters.laplace(seg, mode="constant")[object Object] logger.debug("seg_border: %s", scipy.stats.describe(seg_border, axis=None))[object Object] # logger.debug(str(np.max(seg_border)))[object Object] # logger.debug(str(np.min(seg_border)))[object Object] seg_border[seg_border != 0] = 1[object Object] logger.debug("seg_border: %s", scipy.stats.describe(seg_border, axis=None))[object Object] # scipy.ndimage.morphology.distance_transform_edt[object Object] boundary_dilatation_distance = self.segparams["boundary_dilatation_distance"][object Object] seg = scipy.ndimage.morphology.binary_dilation([object Object] seg_border,[object Object] # seg,[object Object] np.ones([object Object] [[object Object] (boundary_dilatation_distance * 2) + 1,[object Object] (boundary_dilatation_distance * 2) + 1,[object Object] (boundary_dilatation_distance * 2) + 1,[object Object] ][object Object] ),[object Object] )[object Object] if self.keep_temp_properties:[object Object] self.temp_msgc_lowres_discontinuity = seg[object Object] else:[object Object] self.temp_msgc_lowres_discontinuity = None[object Object][object Object] if self.debug_images:[object Object] import sed3[object Object][object Object] pd = sed3.sed3(seg_border) # ), contour=seg)[object Object] pd.show()[object Object] pd = sed3.sed3(seg) # ), contour=seg)[object Object] pd.show()[object Object] # segzoom = scipy.ndimage.interpolation.zoom(seg.astype('float'), zoom,[object Object] # order=0).astype('int8')[object Object] self.stats["t3"] = time.time() - start[object Object] return seg
  ImageGraphCut.__multiscale_gc_lo2hi_run	def __multiscale_gc_lo2hi_run(self): # , pyed):[object Object] """[object Object] Run Graph-Cut segmentation with refinement of low resolution multiscale graph.[object Object] In first step is performed normal GC on low resolution data[object Object] Second step construct finer grid on edges of segmentation from first[object Object] step.[object Object] There is no option for use without [object Object][object Object] """[object Object] # from PyQt4.QtCore import pyqtRemoveInputHook[object Object] # pyqtRemoveInputHook()[object Object] self._msgc_lo2hi_resize_init()[object Object] self.__msgc_step0_init()[object Object][object Object] hard_constraints = self.__msgc_step12_low_resolution_segmentation()[object Object] # ===== high resolution data processing[object Object] seg = self.__msgc_step3_discontinuity_localization()[object Object][object Object] self.stats["t3.1"] = (time.time() - self._start_time)[object Object] graph = Graph([object Object] seg,[object Object] voxelsize=self.voxelsize,[object Object] nsplit=self.segparams["block_size"],[object Object] edge_weight_table=self._msgc_npenalty_table,[object Object] compute_low_nodes_index=True,[object Object] )[object Object][object Object] # graph.run() = graph.generate_base_grid() + graph.split_voxels()[object Object] # graph.run()[object Object] graph.generate_base_grid()[object Object] self.stats["t3.2"] = (time.time() - self._start_time)[object Object] graph.split_voxels()[object Object][object Object] self.stats["t3.3"] = (time.time() - self._start_time)[object Object][object Object] self.stats.update(graph.stats)[object Object] self.stats["t4"] = (time.time() - self._start_time)[object Object] mul_mask, mul_val = self.__msgc_tlinks_area_weight_from_low_segmentation(seg)[object Object] area_weight = 1[object Object] unariesalt = self.__create_tlinks([object Object] self.img,[object Object] self.voxelsize,[object Object] self.seeds,[object Object] area_weight=area_weight,[object Object] hard_constraints=hard_constraints,[object Object] mul_mask=None,[object Object] mul_val=None,[object Object] )[object Object] # N-links prepared[object Object] self.stats["t5"] = (time.time() - self._start_time)[object Object] un, ind = np.unique(graph.msinds, return_index=True)[object Object] self.stats["t6"] = (time.time() - self._start_time)[object Object][object Object] self.stats["t7"] = (time.time() - self._start_time)[object Object] unariesalt2_lo2hi = np.hstack([object Object] [unariesalt[ind, 0, 0].reshape(-1, 1), unariesalt[ind, 0, 1].reshape(-1, 1)][object Object] )[object Object] nlinks_lo2hi = np.hstack([graph.edges, graph.edges_weights.reshape(-1, 1)])[object Object] if self.debug_images:[object Object] import sed3[object Object][object Object] ed = sed3.sed3(unariesalt[:, :, 0].reshape(self.img.shape))[object Object] ed.show()[object Object] import sed3[object Object][object Object] ed = sed3.sed3(unariesalt[:, :, 1].reshape(self.img.shape))[object Object] ed.show()[object Object] # ed = sed3.sed3(seg)[object Object] # ed.show()[object Object] # import sed3[object Object] # ed = sed3.sed3(graph.data)[object Object] # ed.show()[object Object] # import sed3[object Object] # ed = sed3.sed3(graph.msinds)[object Object] # ed.show()[object Object][object Object] # nlinks, unariesalt2, msinds = self.__msgc_step45678_construct_graph(area_weight, hard_constraints, seg)[object Object] # self.__msgc_step9_finish_perform_gc_and_reshape(nlinks, unariesalt2, msinds)[object Object] self.__msgc_step9_finish_perform_gc_and_reshape([object Object] nlinks_lo2hi, unariesalt2_lo2hi, graph.msinds[object Object] )[object Object] self._msgc_lo2hi_resize_clean_finish()
  ImageGraphCut.__multiscale_gc_hi2lo_run	def __multiscale_gc_hi2lo_run(self): # , pyed):[object Object] """[object Object] Run Graph-Cut segmentation with simplifiyng of high resolution multiscale graph.[object Object] In first step is performed normal GC on low resolution data[object Object] Second step construct finer grid on edges of segmentation from first[object Object] step.[object Object] There is no option for use without [object Object][object Object] """[object Object] # from PyQt4.QtCore import pyqtRemoveInputHook[object Object] # pyqtRemoveInputHook()[object Object][object Object] self.__msgc_step0_init()[object Object] hard_constraints = self.__msgc_step12_low_resolution_segmentation()[object Object] # ===== high resolution data processing[object Object] seg = self.__msgc_step3_discontinuity_localization()[object Object] nlinks, unariesalt2, msinds = self.__msgc_step45678_hi2lo_construct_graph([object Object] hard_constraints, seg[object Object] )[object Object] self.__msgc_step9_finish_perform_gc_and_reshape(nlinks, unariesalt2, msinds)

• Loss: MultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Evaluation Dataset

code-search-net/code_search_net

• Dataset: code-search-net/code_search_net
• Size: 15,000 evaluation samples
• Columns: func_name and whole_func_string
• Approximate statistics based on the first 1000 samples:

  	func_name	whole_func_string
  type	string	string
  details	min: 3 tokens mean: 9.23 tokens max: 24 tokens	min: 50 tokens mean: 276.31 tokens max: 384 tokens

• Samples:

  func_name	whole_func_string
  learn	def learn(env,[object Object] network,[object Object] seed=None,[object Object] lr=5e-4,[object Object] total_timesteps=100000,[object Object] buffer_size=50000,[object Object] exploration_fraction=0.1,[object Object] exploration_final_eps=0.02,[object Object] train_freq=1,[object Object] batch_size=32,[object Object] print_freq=100,[object Object] checkpoint_freq=10000,[object Object] checkpoint_path=None,[object Object] learning_starts=1000,[object Object] gamma=1.0,[object Object] target_network_update_freq=500,[object Object] prioritized_replay=False,[object Object] prioritized_replay_alpha=0.6,[object Object] prioritized_replay_beta0=0.4,[object Object] prioritized_replay_beta_iters=None,[object Object] prioritized_replay_eps=1e-6,[object Object] param_noise=False,[object Object] callback=None,[object Object] load_path=None,[object Object] **network_kwargs[object Object] ):[object Object] """Train a deepq model.[object Object][object Object] Parameters[object Object] -------[object Object] env: gym.Env[object Object] environment to train on[object Object] network: string or a function[object Object] neural network to use as a q function approximator. If string, has to be one of the names of registered models in baselines.common.models[object Object] (mlp, cnn, conv_only). If a function, should take an observation tensor and return a latent variable tensor, which[object Object] will be mapped to the Q function heads (see build_q_func in baselines.deepq.models for details on that)[object Object] seed: int or None[object Object] prng seed. The runs with the same seed "should" give the same results. If None, no seeding is used.[object Object] lr: float[object Object] learning rate for adam optimizer[object Object] total_timesteps: int[object Object] number of env steps to optimizer for[object Object] buffer_size: int[object Object] size of the replay buffer[object Object] exploration_fraction: float[object Object] fraction of entire training period over which the exploration rate is annealed[object Object] exploration_final_eps: float[object Object] final value of random action probability[object Object] train_freq: int[object Object] update the model every [object Object] steps.[object Object] set to None to disable printing[object Object] batch_size: int[object Object] size of a batched sampled from replay buffer for training[object Object] print_freq: int[object Object] how often to print out training progress[object Object] set to None to disable printing[object Object] checkpoint_freq: int[object Object] how often to save the model. This is so that the best version is restored[object Object] at the end of the training. If you do not wish to restore the best version at[object Object] the end of the training set this variable to None.[object Object] learning_starts: int[object Object] how many steps of the model to collect transitions for before learning starts[object Object] gamma: float[object Object] discount factor[object Object] target_network_update_freq: int[object Object] update the target network every [object Object] steps.[object Object] prioritized_replay: True[object Object] if True prioritized replay buffer will be used.[object Object] prioritized_replay_alpha: float[object Object] alpha parameter for prioritized replay buffer[object Object] prioritized_replay_beta0: float[object Object] initial value of beta for prioritized replay buffer[object Object] prioritized_replay_beta_iters: int[object Object] number of iterations over which beta will be annealed from initial value[object Object] to 1.0. If set to None equals to total_timesteps.[object Object] prioritized_replay_eps: float[object Object] epsilon to add to the TD errors when updating priorities.[object Object] param_noise: bool[object Object] whether or not to use parameter space noise ([object Object])[object Object] callback: (locals, globals) -> None[object Object] function called at every steps with state of the algorithm.[object Object] If callback returns true training stops.[object Object] load_path: str[object Object] path to load the model from. (default: None)[object Object] **network_kwargs[object Object] additional keyword arguments to pass to the network builder.[object Object][object Object] Returns[object Object] -------[object Object] act: ActWrapper[object Object] Wrapper over act function. Adds ability to save it and load it.[object Object] See header of baselines/deepq/categorical.py for details on the act function.[object Object] """[object Object] # Create all the functions necessary to train the model[object Object][object Object] sess = get_session()[object Object] set_global_seeds(seed)[object Object][object Object] q_func = build_q_func(network, **network_kwargs)[object Object][object Object] # capture the shape outside the closure so that the env object is not serialized[object Object] # by cloudpickle when serializing make_obs_ph[object Object][object Object] observation_space = env.observation_space[object Object] def make_obs_ph(name):[object Object] return ObservationInput(observation_space, name=name)[object Object][object Object] act, train, update_target, debug = deepq.build_train([object Object] make_obs_ph=make_obs_ph,[object Object] q_func=q_func,[object Object] num_actions=env.action_space.n,[object Object] optimizer=tf.train.AdamOptimizer(learning_rate=lr),[object Object] gamma=gamma,[object Object] grad_norm_clipping=10,[object Object] param_noise=param_noise[object Object] )[object Object][object Object] act_params = {[object Object] 'make_obs_ph': make_obs_ph,[object Object] 'q_func': q_func,[object Object] 'num_actions': env.action_space.n,[object Object] }[object Object][object Object] act = ActWrapper(act, act_params)[object Object][object Object] # Create the replay buffer[object Object] if prioritized_replay:[object Object] replay_buffer = PrioritizedReplayBuffer(buffer_size, alpha=prioritized_replay_alpha)[object Object] if prioritized_replay_beta_iters is None:[object Object] prioritized_replay_beta_iters = total_timesteps[object Object] beta_schedule = LinearSchedule(prioritized_replay_beta_iters,[object Object] initial_p=prioritized_replay_beta0,[object Object] final_p=1.0)[object Object] else:[object Object] replay_buffer = ReplayBuffer(buffer_size)[object Object] beta_schedule = None[object Object] # Create the schedule for exploration starting from 1.[object Object] exploration = LinearSchedule(schedule_timesteps=int(exploration_fraction * total_timesteps),[object Object] initial_p=1.0,[object Object] final_p=exploration_final_eps)[object Object][object Object] # Initialize the parameters and copy them to the target network.[object Object] U.initialize()[object Object] update_target()[object Object][object Object] episode_rewards = [0.0][object Object] saved_mean_reward = None[object Object] obs = env.reset()[object Object] reset = True[object Object][object Object] with tempfile.TemporaryDirectory() as td:[object Object] td = checkpoint_path or td[object Object][object Object] model_file = os.path.join(td, "model")[object Object] model_saved = False[object Object][object Object] if tf.train.latest_checkpoint(td) is not None:[object Object] load_variables(model_file)[object Object] logger.log('Loaded model from {}'.format(model_file))[object Object] model_saved = True[object Object] elif load_path is not None:[object Object] load_variables(load_path)[object Object] logger.log('Loaded model from {}'.format(load_path))[object Object][object Object][object Object] for t in range(total_timesteps):[object Object] if callback is not None:[object Object] if callback(locals(), globals()):[object Object] break[object Object] # Take action and update exploration to the newest value[object Object] kwargs = {}[object Object] if not param_noise:[object Object] update_eps = exploration.value(t)[object Object] update_param_noise_threshold = 0.[object Object] else:[object Object] update_eps = 0.[object Object] # Compute the threshold such that the KL divergence between perturbed and non-perturbed[object Object] # policy is comparable to eps-greedy exploration with eps = exploration.value(t).[object Object] # See Appendix C.1 in Parameter Space Noise for Exploration, Plappert et al., 2017[object Object] # for detailed explanation.[object Object] update_param_noise_threshold = -np.log(1. - exploration.value(t) + exploration.value(t) / float(env.action_space.n))[object Object] kwargs['reset'] = reset[object Object] kwargs['update_param_noise_threshold'] = update_param_noise_threshold[object Object] kwargs['update_param_noise_scale'] = True[object Object] action = act(np.array(obs)[None], update_eps=update_eps, **kwargs)[0][object Object] env_action = action[object Object] reset = False[object Object] new_obs, rew, done, _ = env.step(env_action)[object Object] # Store transition in the replay buffer.[object Object] replay_buffer.add(obs, action, rew, new_obs, float(done))[object Object] obs = new_obs[object Object][object Object] episode_rewards[-1] += rew[object Object] if done:[object Object] obs = env.reset()[object Object] episode_rewards.append(0.0)[object Object] reset = True[object Object][object Object] if t > learning_starts and t % train_freq == 0:[object Object] # Minimize the error in Bellman's equation on a batch sampled from replay buffer.[object Object] if prioritized_replay:[object Object] experience = replay_buffer.sample(batch_size, beta=beta_schedule.value(t))[object Object] (obses_t, actions, rewards, obses_tp1, dones, weights, batch_idxes) = experience[object Object] else:[object Object] obses_t, actions, rewards, obses_tp1, dones = replay_buffer.sample(batch_size)[object Object] weights, batch_idxes = np.ones_like(rewards), None[object Object] td_errors = train(obses_t, actions, rewards, obses_tp1, dones, weights)[object Object] if prioritized_replay:[object Object] new_priorities = np.abs(td_errors) + prioritized_replay_eps[object Object] replay_buffer.update_priorities(batch_idxes, new_priorities)[object Object][object Object] if t > learning_starts and t % target_network_update_freq == 0:[object Object] # Update target network periodically.[object Object] update_target()[object Object][object Object] mean_100ep_reward = round(np.mean(episode_rewards[-101:-1]), 1)[object Object] num_episodes = len(episode_rewards)[object Object] if done and print_freq is not None and len(episode_rewards) % print_freq == 0:[object Object] logger.record_tabular("steps", t)[object Object] logger.record_tabular("episodes", num_episodes)[object Object] logger.record_tabular("mean 100 episode reward", mean_100ep_reward)[object Object] logger.record_tabular("% time spent exploring", int(100 * exploration.value(t)))[object Object] logger.dump_tabular()[object Object][object Object] if (checkpoint_freq is not None and t > learning_starts and[object Object] num_episodes > 100 and t % checkpoint_freq == 0):[object Object] if saved_mean_reward is None or mean_100ep_reward > saved_mean_reward:[object Object] if print_freq is not None:[object Object] logger.log("Saving model due to mean reward increase: {} -> {}".format([object Object] saved_mean_reward, mean_100ep_reward))[object Object] save_variables(model_file)[object Object] model_saved = True[object Object] saved_mean_reward = mean_100ep_reward[object Object] if model_saved:[object Object] if print_freq is not None:[object Object] logger.log("Restored model with mean reward: {}".format(saved_mean_reward))[object Object] load_variables(model_file)[object Object][object Object] return act
  ActWrapper.save_act	def save_act(self, path=None):[object Object] """Save model to a pickle located at [object Object]"""[object Object] if path is None:[object Object] path = os.path.join(logger.get_dir(), "model.pkl")[object Object][object Object] with tempfile.TemporaryDirectory() as td:[object Object] save_variables(os.path.join(td, "model"))[object Object] arc_name = os.path.join(td, "packed.zip")[object Object] with zipfile.ZipFile(arc_name, 'w') as zipf:[object Object] for root, dirs, files in os.walk(td):[object Object] for fname in files:[object Object] file_path = os.path.join(root, fname)[object Object] if file_path != arc_name:[object Object] zipf.write(file_path, os.path.relpath(file_path, td))[object Object] with open(arc_name, "rb") as f:[object Object] model_data = f.read()[object Object] with open(path, "wb") as f:[object Object] cloudpickle.dump((model_data, self._act_params), f)
  nature_cnn	def nature_cnn(unscaled_images, **conv_kwargs):[object Object] """[object Object] CNN from Nature paper.[object Object] """[object Object] scaled_images = tf.cast(unscaled_images, tf.float32) / 255.[object Object] activ = tf.nn.relu[object Object] h = activ(conv(scaled_images, 'c1', nf=32, rf=8, stride=4, init_scale=np.sqrt(2),[object Object] **conv_kwargs))[object Object] h2 = activ(conv(h, 'c2', nf=64, rf=4, stride=2, init_scale=np.sqrt(2), **conv_kwargs))[object Object] h3 = activ(conv(h2, 'c3', nf=64, rf=3, stride=1, init_scale=np.sqrt(2), **conv_kwargs))[object Object] h3 = conv_to_fc(h3)[object Object] return activ(fc(h3, 'fc1', nh=512, init_scale=np.sqrt(2)))

• Loss: MultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.42.3
• PyTorch: 2.1.2
• Accelerate: 0.32.1
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

MultipleNegativesRankingLoss