This is a sentence-transformers model finetuned from Shuu12121/CodeModernBERT-Owl-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Shuu12121/CodeModernBERT-Owl-v1
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '#\nDeletes the cluster, including the Kubernetes endpoint and all worker\nnodes.\n\nFirewalls and routes that were configured during cluster creation\nare also deleted.\n\nOther Google Compute Engine resources that might be in use by the cluster,\nsuch as load balancer resources, are not deleted if they weren\'t present\nwhen the cluster was initially created.\n\n@overload delete_cluster(request, options = nil)\nPass arguments to `delete_cluster` via a request object, either of type\n{::Google::Cloud::Container::V1::DeleteClusterRequest} or an equivalent Hash.\n\n@param request [::Google::Cloud::Container::V1::DeleteClusterRequest, ::Hash]\nA request object representing the call parameters. Required. To specify no\nparameters, or to keep all the default parameter values, pass an empty Hash.\n@param options [::Gapic::CallOptions, ::Hash]\nOverrides the default settings for this call, e.g, timeout, retries, etc. Optional.\n\n@overload delete_cluster(project_id: nil, zone: nil, cluster_id: nil, name: nil)\nPass arguments to `delete_cluster` via keyword arguments. Note that at\nleast one keyword argument is required. To specify no parameters, or to keep all\nthe default parameter values, pass an empty Hash as a request object (see above).\n\n@param project_id [::String]\nDeprecated. The Google Developers Console [project ID or project\nnumber](https://cloud.google.com/resource-manager/docs/creating-managing-projects).\nThis field has been deprecated and replaced by the name field.\n@param zone [::String]\nDeprecated. The name of the Google Compute Engine\n[zone](https://cloud.google.com/compute/docs/zones#available) in which the\ncluster resides. This field has been deprecated and replaced by the name\nfield.\n@param cluster_id [::String]\nDeprecated. The name of the cluster to delete.\nThis field has been deprecated and replaced by the name field.\n@param name [::String]\nThe name (project, location, cluster) of the cluster to delete.\nSpecified in the format `projects/*/locations/*/clusters/*`.\n\n@yield [response, operation] Access the result along with the RPC operation\n@yieldparam response [::Google::Cloud::Container::V1::Operation]\n@yieldparam operation [::GRPC::ActiveCall::Operation]\n\n@return [::Google::Cloud::Container::V1::Operation]\n\n@raise [::Google::Cloud::Error] if the RPC is aborted.\n\n@example Basic example\nrequire "google/cloud/container/v1"\n\n# Create a client object. The client can be reused for multiple calls.\nclient = Google::Cloud::Container::V1::ClusterManager::Client.new\n\n# Create a request. To set request fields, pass in keyword arguments.\nrequest = Google::Cloud::Container::V1::DeleteClusterRequest.new\n\n# Call the delete_cluster method.\nresult = client.delete_cluster request\n\n# The returned object is of type Google::Cloud::Container::V1::Operation.\np result',
    'def delete_cluster request, options = nil\n              raise ::ArgumentError, "request must be provided" if request.nil?\n\n              request = ::Gapic::Protobuf.coerce request, to: ::Google::Cloud::Container::V1::DeleteClusterRequest\n\n              # Converts hash and nil to an options object\n              options = ::Gapic::CallOptions.new(**options.to_h) if options.respond_to? :to_h\n\n              # Customize the options with defaults\n              metadata = @config.rpcs.delete_cluster.metadata.to_h\n\n              # Set x-goog-api-client, x-goog-user-project and x-goog-api-version headers\n              metadata[:"x-goog-api-client"] ||= ::Gapic::Headers.x_goog_api_client \\\n                lib_name: @config.lib_name, lib_version: @config.lib_version,\n                gapic_version: ::Google::Cloud::Container::V1::VERSION\n              metadata[:"x-goog-api-version"] = API_VERSION unless API_VERSION.empty?\n              metadata[:"x-goog-user-project"] = @quota_project_id if @quota_project_id\n\n              header_params = {}\n              if request.name\n                header_params["name"] = request.name\n              end\n\n              request_params_header = header_params.map { |k, v| "#{k}=#{v}" }.join("&")\n              metadata[:"x-goog-request-params"] ||= request_params_header\n\n              options.apply_defaults timeout:      @config.rpcs.delete_cluster.timeout,\n                                     metadata:     metadata,\n                                     retry_policy: @config.rpcs.delete_cluster.retry_policy\n\n              options.apply_defaults timeout:      @config.timeout,\n                                     metadata:     @config.metadata,\n                                     retry_policy: @config.retry_policy\n\n              @cluster_manager_stub.call_rpc :delete_cluster, request, options: options do |response, operation|\n                yield response, operation if block_given?\n              end\n            rescue ::GRPC::BadStatus => e\n              raise ::Google::Cloud::Error.from_error(e)\n            end',
    'device(deviceType, deviceId = 0) {\n\t        return new DLDevice(deviceType, deviceId, this.lib);\n\t    }',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 3,999,600 training samples
Columns: sentence_0, sentence_1, and label
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label
type string string float
details
min: 8 tokens
mean: 74.13 tokens
max: 1024 tokens
min: 13 tokens
mean: 154.33 tokens
max: 1024 tokens
min: 1.0
mean: 1.0
max: 1.0

Samples:

sentence_0	sentence_1	label
`Set the column title[object Object][object Object]@param column - column number (first column is: 0)[object Object]@param title - new column title`	`setHeader = function(column, newValue) {[object Object] const obj = this;[object Object][object Object] if (obj.headers[column]) {[object Object] const oldValue = obj.headers[column].textContent;[object Object] const onchangeheaderOldValue = (obj.options.columns && obj.options.columns[column] && obj.options.columns[column].title)`
Elsewhere this is known as a "Weak Value Map". Whereas a std JS WeakMap[object Object]is weak on its keys, this map is weak on its values. It does not retain these[object Object]values strongly. If a given value disappears, then the entries for it[object Object]disappear from every weak-value-map that holds it as a value.[object Object][object Object]Just as a WeakMap only allows gc-able values as keys, a weak-value-map[object Object]only allows gc-able values as values.[object Object][object Object]Unlike a WeakMap, a weak-value-map unavoidably exposes the non-determinism of[object Object]gc to its clients. Thus, both the ability to create one, as well as each[object Object]created one, must be treated as dangerous capabilities that must be closely[object Object]held. A program with access to these can read side channels though gc that do[object Object]not* rely on the ability to measure duration. This is a separate, and bad,[object Object]timing-independent side channel.[object Object][object Object]This non-determinism also enables code to escape deterministic replay. In a[object Object]blockchain context, this could cause validators to differ from each other,[object Object]preventing consensus, and thus preventing ...	`makeFinalizingMap = (finalizer, opts) => {[object Object] const { weakValues = false } = opts`
Creates a function that memoizes the result of [object Object]. If [object Object] is[object Object]provided, it determines the cache key for storing the result based on the[object Object]arguments provided to the memoized function. By default, the first argument[object Object]provided to the memoized function is used as the map cache key. The [object Object][object Object]is invoked with the [object Object] binding of the memoized function.[object Object][object Object][object Object] The cache is exposed as the [object Object] property on the memoized[object Object]function. Its creation may be customized by replacing the [object Object][object Object]constructor with one whose instances implement the[object Object][object Object][object Object]method interface of [object Object], [object Object], [object Object], and [object Object].[object Object][object Object]@static[object Object]@memberOf _[object Object]@since 0.1.0[object Object]@category Function[object Object]@param {Function} func The function to have its output memoized.[object Object]@param {Function} [resolver] The function to resolve the cache key.[object Object]@returns {Function} Returns the new memoized function.[object Object]@example[object Object][object Object]var object = { 'a': 1, 'b': 2 };[object Object]var othe...	`function memoize(func, resolver) {[object Object] if (typeof func != 'function'`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 150
per_device_eval_batch_size: 150
num_train_epochs: 1
fp16: True
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 150
per_device_eval_batch_size: 150
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.0188	500	0.2957
0.0375	1000	0.1174
0.0563	1500	0.1148
0.0750	2000	0.104
0.0938	2500	0.0977
0.1125	3000	0.0944
0.1313	3500	0.0885
0.1500	4000	0.083
0.1688	4500	0.0817
0.1875	5000	0.077
0.2063	5500	0.0764
0.2250	6000	0.0725
0.2438	6500	0.0698
0.2625	7000	0.0663
0.2813	7500	0.0644
0.3000	8000	0.0606
0.3188	8500	0.0587
0.3375	9000	0.0596
0.3563	9500	0.0566
0.3750	10000	0.0536
0.3938	10500	0.0514
0.4125	11000	0.0532
0.4313	11500	0.0501
0.4500	12000	0.0478
0.4688	12500	0.0483
0.4875	13000	0.0461
0.5063	13500	0.0444
0.5251	14000	0.0443
0.5438	14500	0.0402
0.5626	15000	0.0417
0.5813	15500	0.0386
0.6001	16000	0.0421
0.6188	16500	0.0368
0.6376	17000	0.036
0.6563	17500	0.0352
0.6751	18000	0.0339
0.6938	18500	0.0336
0.7126	19000	0.0334
0.7313	19500	0.0312
0.7501	20000	0.0325
0.7688	20500	0.0317
0.7876	21000	0.0284
0.8063	21500	0.0281
0.8251	22000	0.0294
0.8438	22500	0.0283
0.8626	23000	0.0277
0.8813	23500	0.0268
0.9001	24000	0.0254
0.9188	24500	0.0249
0.9376	25000	0.0255
0.9563	25500	0.0251
0.9751	26000	0.0244
0.9938	26500	0.0249

Framework Versions

Python: 3.11.13
Sentence Transformers: 4.1.0
Transformers: 4.53.2
PyTorch: 2.6.0+cu124
Accelerate: 1.9.0
Datasets: 3.6.0
Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss