This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the magpie-qwen2.5-coder-pro-300_k-query-positive-pair dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Alibaba-NLP/gte-modernbert-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- magpie-qwen2.5-coder-pro-300_k-query-positive-pair
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("lemon-mint/gte-modernbert-base-code-3")
# Run inference
sentences = [
    'Design a simple Python script to backup a database using SQL commands and save it to a file. This script should use PostgreSQL as the database management system.',
    'import subprocess\nimport os\nfrom datetime import datetime\n\ndef backup_database(db_name, db_user, db_host, db_port, backup_dir):\n    # Ensure the backup directory exists\n    if not os.path.exists(backup_dir):\n        os.makedirs(backup_dir)\n\n    # Create a timestamp for the backup file\n    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")\n    backup_file = os.path.join(backup_dir, f"{db_name}_backup_{timestamp}.sql")\n\n    # Construct the pg_dump command\n    command = [\n        "pg_dump",\n        "-U", db_user,\n        "-h", db_host,\n        "-p", str(db_port),\n        "-F", "c",  # Custom format, which is compressed\n        "-b",       # Include large objects\n        "-v",       # Verbose mode\n        "-f", backup_file,\n        db_name\n    ]\n\n    try:\n        # Run the pg_dump command\n        subprocess.run(command, check=True)\n        print(f"Backup successful: {backup_file}")\n    except subprocess.CalledProcessError as e:\n        print(f"An error occurred while backing up the database: {e}")\n\nif __name__ == "__main__":\n    # Database connection details\n    db_name = "your_database_name"\n    db_user = "your_database_user"\n    db_host = "localhost"  # or your database host\n    db_port = 5432         # default PostgreSQL port\n    backup_dir = "/path/to/your/backup/directory"\n\n    # Call the backup function\n    backup_database(db_name, db_user, db_host, db_port, backup_dir)\n',
    'def fibonacci_sequence(n):\n    # Initialize the first two Fibonacci numbers\n    fib_sequence = [0, 1]\n    \n    # Generate the Fibonacci sequence up to n\n    while True:\n        next_fib = fib_sequence[-1] + fib_sequence[-2]\n        if next_fib > n:\n            break\n        fib_sequence.append(next_fib)\n    \n    # If n is less than 1, return the sequence up to n\n    if n < 1:\n        return fib_sequence[:n+1]\n    \n    return fib_sequence\n\n# Example usage\nn = int(input("Enter a number to generate the Fibonacci sequence up to: "))\nsequence = fibonacci_sequence(n)\nprint(f"Fibonacci sequence up to {n}: {sequence}")\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

magpie-qwen2.5-coder-pro-300_k-query-positive-pair

Dataset: magpie-qwen2.5-coder-pro-300_k-query-positive-pair at 1c3e977
Size: 78,926 training samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 8 tokens
mean: 35.38 tokens
max: 184 tokens
min: 3 tokens
mean: 235.42 tokens
max: 1097 tokens

Samples:

query	positive
`Can you provide a Python script that reads a CSV file and performs basic data analysis on it? The CSV file contains sales data with columns like Date, Product, Quantity, and Price. The analysis should include:`	import pandas as pd[object Object][object Object]# Load the CSV file into a DataFrame[object Object]def load_data(file_path):[object Object] try:[object Object] data = pd.read_csv(file_path)[object Object] return data[object Object] except FileNotFoundError:[object Object] print(f"Error: The file {file_path} was not found.")[object Object] return None[object Object] except pd.errors.EmptyDataError:[object Object] print("Error: The file is empty.")[object Object] return None[object Object] except pd.errors.ParserError:[object Object] print("Error: There was a problem parsing the file.")[object Object] return None[object Object][object Object]# Perform basic data analysis[object Object]def analyze_data(data):[object Object] if data is None:[object Object] return[object Object][object Object] # Display the first few rows of the DataFrame[object Object] print("First few rows of the dataset:")[object Object] print(data.head())[object Object][object Object] # Get a summary of the DataFrame[object Object] print("\nSummary of the dataset:")[object Object] print(data.info())[object Object][object Object] # Get basic statistics of the DataFrame[object Object] print("\nBasic statistics of the dataset:")[object Object] print(data.describe())[object Object][object Object] # Calculate total sales[object Object] data['Total Sales'] = data['Quantity'] * data['Price'][object Object] t...
`How can I retrieve the virtual machine instances using the Alibaba Cloud SDK for Java?`	import com.aliyuncs.DefaultAcsClient;[object Object]import com.aliyuncs.IAcsClient;[object Object]import com.aliyuncs.ecs.model.v20140526.DescribeInstancesRequest;[object Object]import com.aliyuncs.ecs.model.v20140526.DescribeInstancesResponse;[object Object]import com.aliyuncs.exceptions.ClientException;[object Object]import com.aliyuncs.profile.DefaultProfile;[object Object][object Object]public class DescribeInstancesExample {[object Object] public static void main(String[] args) {[object Object] // Step 1: Configure your Alibaba Cloud credentials[object Object] String accessKeyId = "yourAccessKeyId";[object Object] String accessKeySecret = "yourAccessKeySecret";[object Object] String regionId = "yourRegionId"; // e.g., "cn-hangzhou"[object Object][object Object] // Step 2: Create a profile and initialize the client[object Object] DefaultProfile profile = DefaultProfile.getProfile(regionId, accessKeyId, accessKeySecret);[object Object] IAcsClient client = new DefaultAcsClient(profile);[object Object][object Object] // Step 3: Create a request to describe instances[object Object] DescribeInstancesRequest request = new DescribeInstancesRequest();[object Object] request.setPageSize(10); /...
`I have a list of product reviews where the titles and body text of each review are separated by a period. How can I split these reviews into their respective titles and bodies in Python?`	# Sample list of reviews[object Object]reviews = [[object Object] "Great product. I really enjoyed using it!",[object Object] "Not what I expected. It broke after a week.",[object Object] "Excellent quality. Very satisfied with the purchase."[object Object]][object Object][object Object]# Function to split reviews into titles and bodies[object Object]def split_reviews(reviews):[object Object] split_reviews = [][object Object] for review in reviews:[object Object] # Split the review into title and body[object Object] parts = review.split('.', 1) # Split only at the first period[object Object] if len(parts) == 2:[object Object] title = parts[0].strip() # Remove any leading/trailing whitespace[object Object] body = parts[1].strip() # Remove any leading/trailing whitespace[object Object] split_reviews.append((title, body))[object Object] else:[object Object] # Handle cases where there is no period in the review[object Object] split_reviews.append((parts[0].strip(), ''))[object Object] return split_reviews[object Object][object Object]# Split the reviews[object Object]split_reviews = split_reviews(reviews)[object Object][object Object]# Print the results[object Object]for title, body in split_reviews:[object Object] print(f"Title: {title}")[object Object] print(f"Bo...

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 64
learning_rate: 2e-05
num_train_epochs: 2
warmup_ratio: 0.1
bf16: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0081	10	0.2042
0.0162	20	0.1868
0.0243	30	0.1609
0.0324	40	0.1735
0.0405	50	0.1792
0.0486	60	0.1601
0.0567	70	0.1272
0.0648	80	0.1166
0.0729	90	0.106
0.0810	100	0.1165
0.0891	110	0.0952
0.0972	120	0.1117
0.1053	130	0.1172
0.1135	140	0.0666
0.1216	150	0.0749
0.1297	160	0.0627
0.1378	170	0.1283
0.1459	180	0.0876
0.1540	190	0.0825
0.1621	200	0.0666
0.1702	210	0.0796
0.1783	220	0.0612
0.1864	230	0.077
0.1945	240	0.0625
0.2026	250	0.064
0.2107	260	0.0692
0.2188	270	0.0461
0.2269	280	0.0555
0.2350	290	0.0959
0.2431	300	0.0716
0.2512	310	0.0542
0.2593	320	0.0667
0.2674	330	0.0455
0.2755	340	0.0773
0.2836	350	0.0538
0.2917	360	0.0635
0.2998	370	0.0782
0.3079	380	0.0584
0.3160	390	0.0657
0.3241	400	0.0637
0.3323	410	0.0498
0.3404	420	0.0569
0.3485	430	0.0725
0.3566	440	0.0452
0.3647	450	0.0778
0.3728	460	0.0529
0.3809	470	0.0596
0.3890	480	0.0512
0.3971	490	0.0492
0.4052	500	0.0635
0.4133	510	0.0513
0.4214	520	0.0683
0.4295	530	0.0819
0.4376	540	0.069
0.4457	550	0.0573
0.4538	560	0.0638
0.4619	570	0.0937
0.4700	580	0.0468
0.4781	590	0.0596
0.4862	600	0.0698
0.4943	610	0.0684
0.5024	620	0.0467
0.5105	630	0.0547
0.5186	640	0.0435
0.5267	650	0.0435
0.5348	660	0.074
0.5429	670	0.0673
0.5511	680	0.0528
0.5592	690	0.0398
0.5673	700	0.0518
0.5754	710	0.0337
0.5835	720	0.0429
0.5916	730	0.0524
0.5997	740	0.0527
0.6078	750	0.0508
0.6159	760	0.0405
0.6240	770	0.0395
0.6321	780	0.0766
0.6402	790	0.0653
0.6483	800	0.076
0.6564	810	0.055
0.6645	820	0.0455
0.6726	830	0.0563
0.6807	840	0.0511
0.6888	850	0.0277
0.6969	860	0.0369
0.7050	870	0.0427
0.7131	880	0.054
0.7212	890	0.0494
0.7293	900	0.047
0.7374	910	0.0335
0.7455	920	0.0495
0.7536	930	0.0618
0.7618	940	0.0552
0.7699	950	0.0444
0.7780	960	0.0336
0.7861	970	0.042
0.7942	980	0.0519
0.8023	990	0.0415
0.8104	1000	0.037
0.8185	1010	0.0718
0.8266	1020	0.0307
0.8347	1030	0.0573
0.8428	1040	0.0509
0.8509	1050	0.0375
0.8590	1060	0.0352
0.8671	1070	0.0338
0.8752	1080	0.0644
0.8833	1090	0.0392
0.8914	1100	0.0433
0.8995	1110	0.0507
0.9076	1120	0.0439
0.9157	1130	0.0322
0.9238	1140	0.0505
0.9319	1150	0.0403
0.9400	1160	0.0482
0.9481	1170	0.0502
0.9562	1180	0.0353
0.9643	1190	0.0346
0.9724	1200	0.0526
0.9806	1210	0.0476
0.9887	1220	0.0336
0.9968	1230	0.0376
1.0049	1240	0.0216
1.0130	1250	0.0314
1.0211	1260	0.031
1.0292	1270	0.0255
1.0373	1280	0.0331
1.0454	1290	0.032
1.0535	1300	0.0443
1.0616	1310	0.0252
1.0697	1320	0.0242
1.0778	1330	0.0281
1.0859	1340	0.0305
1.0940	1350	0.0278
1.1021	1360	0.0276
1.1102	1370	0.0257
1.1183	1380	0.03
1.1264	1390	0.0299
1.1345	1400	0.0423
1.1426	1410	0.0494
1.1507	1420	0.0338
1.1588	1430	0.0284
1.1669	1440	0.0311
1.1750	1450	0.0345
1.1831	1460	0.0415
1.1912	1470	0.0223
1.1994	1480	0.0367
1.2075	1490	0.0317
1.2156	1500	0.0331
1.2237	1510	0.044
1.2318	1520	0.0398
1.2399	1530	0.023
1.2480	1540	0.0249
1.2561	1550	0.0241
1.2642	1560	0.0357
1.2723	1570	0.0358
1.2804	1580	0.0244
1.2885	1590	0.0422
1.2966	1600	0.0213
1.3047	1610	0.0371
1.3128	1620	0.0346
1.3209	1630	0.0313
1.3290	1640	0.0192
1.3371	1650	0.0263
1.3452	1660	0.0277
1.3533	1670	0.0301
1.3614	1680	0.0375
1.3695	1690	0.0227
1.3776	1700	0.0213
1.3857	1710	0.022
1.3938	1720	0.0426
1.4019	1730	0.025
1.4100	1740	0.026
1.4182	1750	0.0487
1.4263	1760	0.0359
1.4344	1770	0.0285
1.4425	1780	0.0384
1.4506	1790	0.0388
1.4587	1800	0.0304
1.4668	1810	0.0321
1.4749	1820	0.0183
1.4830	1830	0.0178
1.4911	1840	0.0311
1.4992	1850	0.0243
1.5073	1860	0.0358
1.5154	1870	0.0274
1.5235	1880	0.034
1.5316	1890	0.0255
1.5397	1900	0.027
1.5478	1910	0.023
1.5559	1920	0.0251
1.5640	1930	0.0376
1.5721	1940	0.032
1.5802	1950	0.0432
1.5883	1960	0.0307
1.5964	1970	0.0282
1.6045	1980	0.0227
1.6126	1990	0.0221
1.6207	2000	0.018
1.6288	2010	0.0189
1.6370	2020	0.0417
1.6451	2030	0.0244
1.6532	2040	0.0258
1.6613	2050	0.0488
1.6694	2060	0.0322
1.6775	2070	0.0348
1.6856	2080	0.0202
1.6937	2090	0.023
1.7018	2100	0.0298
1.7099	2110	0.0391
1.7180	2120	0.019
1.7261	2130	0.0294
1.7342	2140	0.0268
1.7423	2150	0.0308
1.7504	2160	0.032
1.7585	2170	0.0287
1.7666	2180	0.0452
1.7747	2190	0.0425
1.7828	2200	0.0294
1.7909	2210	0.0227
1.7990	2220	0.0183
1.8071	2230	0.0158
1.8152	2240	0.0149
1.8233	2250	0.0347
1.8314	2260	0.0129
1.8395	2270	0.0305
1.8476	2280	0.0173
1.8558	2290	0.0357
1.8639	2300	0.0195
1.8720	2310	0.0392
1.8801	2320	0.0168
1.8882	2330	0.0266
1.8963	2340	0.0143
1.9044	2350	0.0296
1.9125	2360	0.0367
1.9206	2370	0.0151
1.9287	2380	0.0253
1.9368	2390	0.0415
1.9449	2400	0.0371
1.9530	2410	0.0267
1.9611	2420	0.0327
1.9692	2430	0.0378
1.9773	2440	0.0267
1.9854	2450	0.0275
1.9935	2460	0.0263

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.4.0
Transformers: 4.48.1
PyTorch: 2.1.0+cu118
Accelerate: 1.3.0
Datasets: 3.2.0
Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	query	positive
type	string	string
details	min: 8 tokens mean: 35.38 tokens max: 184 tokens	min: 3 tokens mean: 235.42 tokens max: 1097 tokens