This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: intfloat/multilingual-e5-large
Maximum Sequence Length: 64 tokens
Output Dimensionality: 1024 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 64, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Ruslan10/fine-tuned-e5-large")
# Run inference
sentences = [
    'query: aston martin v8 vantage 5 3 kabriolet avtomaticheskaja benzin dverej 2 mest 4',
    'passage: aston martin v8 vantage',
    'passage: alfa romeo stelvio',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 66,405 training samples
Columns: anchor and positive
Approximate statistics based on the first 1000 samples:
anchor positive
type string string
details
min: 8 tokens
mean: 25.5 tokens
max: 40 tokens
min: 6 tokens
mean: 8.46 tokens
max: 13 tokens

Samples:

anchor	positive
`query: ac cobra 4 9i rodster mehanicheskaja benzin dverej 2 mest 2`	`passage: ac cobra`
`query: ac cobra 6 2i rodster mehanicheskaja benzin dverej 2 mest 2`	`passage: ac cobra`
`query: ac ace 2 9i rodster mehanicheskaja benzin dverej 2 mest 2`	`passage: ac ace`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 16
learning_rate: 1e-05
num_train_epochs: 2
warmup_ratio: 0.1
fp16: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 1e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0048	10	2.7309
0.0096	20	2.6547
0.0145	30	2.4704
0.0193	40	2.1369
0.0241	50	1.6635
0.0289	60	0.954
0.0337	70	0.4696
0.0385	80	0.2474
0.0434	90	0.1549
0.0482	100	0.0978
0.0530	110	0.062
0.0578	120	0.0824
0.0626	130	0.0674
0.0674	140	0.0824
0.0723	150	0.0539
0.0771	160	0.0599
0.0819	170	0.0603
0.0867	180	0.0987
0.0915	190	0.0358
0.0963	200	0.0656
0.1012	210	0.0594
0.1060	220	0.071
0.1108	230	0.0522
0.1156	240	0.0452
0.1204	250	0.0369
0.1252	260	0.0384
0.1301	270	0.0313
0.1349	280	0.0569
0.1397	290	0.0606
0.1445	300	0.0519
0.1493	310	0.0443
0.1541	320	0.0348
0.1590	330	0.0358
0.1638	340	0.0756
0.1686	350	0.0666
0.1734	360	0.0571
0.1782	370	0.0329
0.1830	380	0.0689
0.1879	390	0.0292
0.1927	400	0.0465
0.1975	410	0.0777
0.2023	420	0.068
0.2071	430	0.0473
0.2119	440	0.0709
0.2168	450	0.0401
0.2216	460	0.0451
0.2264	470	0.0687
0.2312	480	0.0679
0.2360	490	0.0736
0.2408	500	0.0521
0.2457	510	0.0724
0.2505	520	0.0449
0.2553	530	0.046
0.2601	540	0.0697
0.2649	550	0.0415
0.2697	560	0.049
0.2746	570	0.0375
0.2794	580	0.047
0.2842	590	0.039
0.2890	600	0.0681
0.2938	610	0.0473
0.2987	620	0.0721
0.3035	630	0.0416
0.3083	640	0.0404
0.3131	650	0.058
0.3179	660	0.1074
0.3227	670	0.0623
0.3276	680	0.0978
0.3324	690	0.0783
0.3372	700	0.0417
0.3420	710	0.0752
0.3468	720	0.0463
0.3516	730	0.0351
0.3565	740	0.0421
0.3613	750	0.0575
0.3661	760	0.0406
0.3709	770	0.0284
0.3757	780	0.0361
0.3805	790	0.0508
0.3854	800	0.0472
0.3902	810	0.0809
0.3950	820	0.0446
0.3998	830	0.0495
0.4046	840	0.0646
0.4094	850	0.0541
0.4143	860	0.0487
0.4191	870	0.0643
0.4239	880	0.0483
0.4287	890	0.076
0.4335	900	0.0486
0.4383	910	0.0672
0.4432	920	0.0794
0.4480	930	0.0408
0.4528	940	0.0349
0.4576	950	0.0661
0.4624	960	0.0518
0.4672	970	0.0999
0.4721	980	0.0601
0.4769	990	0.092
0.4817	1000	0.0638
0.4865	1010	0.05
0.4913	1020	0.0644
0.4961	1030	0.0848
0.5010	1040	0.0396
0.5058	1050	0.0652
0.5106	1060	0.0715
0.5154	1070	0.0884
0.5202	1080	0.0378
0.5250	1090	0.0812
0.5299	1100	0.0464
0.5347	1110	0.0565
0.5395	1120	0.0509
0.5443	1130	0.0694
0.5491	1140	0.0376
0.5539	1150	0.0413
0.5588	1160	0.0478
0.5636	1170	0.0577
0.5684	1180	0.0556
0.5732	1190	0.0284
0.5780	1200	0.0346
0.5829	1210	0.0674
0.5877	1220	0.0503
0.5925	1230	0.0925
0.5973	1240	0.0427
0.6021	1250	0.0482
0.6069	1260	0.0486
0.6118	1270	0.0237
0.6166	1280	0.0444
0.6214	1290	0.0598
0.6262	1300	0.0557
0.6310	1310	0.0512
0.6358	1320	0.043
0.6407	1330	0.0637
0.6455	1340	0.0701
0.6503	1350	0.0553
0.6551	1360	0.0628
0.6599	1370	0.0596
0.6647	1380	0.0602
0.6696	1390	0.0622
0.6744	1400	0.0589
0.6792	1410	0.0906
0.6840	1420	0.0349
0.6888	1430	0.0694
0.6936	1440	0.0577
0.6985	1450	0.0556
0.7033	1460	0.052
0.7081	1470	0.0294
0.7129	1480	0.0633
0.7177	1490	0.0617
0.7225	1500	0.0504
0.7274	1510	0.052
0.7322	1520	0.0536
0.7370	1530	0.0402
0.7418	1540	0.041
0.7466	1550	0.0852
0.7514	1560	0.0735
0.7563	1570	0.051
0.7611	1580	0.0718
0.7659	1590	0.0562
0.7707	1600	0.0374
0.7755	1610	0.0559
0.7803	1620	0.0286
0.7852	1630	0.0218
0.7900	1640	0.0519
0.7948	1650	0.0502
0.7996	1660	0.0787
0.8044	1670	0.0469
0.8092	1680	0.0365
0.8141	1690	0.0535
0.8189	1700	0.0408
0.8237	1710	0.038
0.8285	1720	0.0329
0.8333	1730	0.0593
0.8382	1740	0.0582
0.8430	1750	0.0585
0.8478	1760	0.0688
0.8526	1770	0.0581
0.8574	1780	0.0519
0.8622	1790	0.0466
0.8671	1800	0.0552
0.8719	1810	0.0448
0.8767	1820	0.0373
0.8815	1830	0.0259
0.8863	1840	0.0416
0.8911	1850	0.046
0.8960	1860	0.0432
0.9008	1870	0.0671
0.9056	1880	0.0636
0.9104	1890	0.048
0.9152	1900	0.0348
0.9200	1910	0.056
0.9249	1920	0.0421
0.9297	1930	0.0752
0.9345	1940	0.0342
0.9393	1950	0.0452
0.9441	1960	0.0598
0.9489	1970	0.0454
0.9538	1980	0.068
0.9586	1990	0.066
0.9634	2000	0.0513
0.9682	2010	0.0477
0.9730	2020	0.0836
0.9778	2030	0.0315
0.9827	2040	0.0611
0.9875	2050	0.0539
0.9923	2060	0.0332
0.9971	2070	0.0543
1.0019	2080	0.0145
1.0067	2090	0.0584
1.0116	2100	0.0562
1.0164	2110	0.0458
1.0212	2120	0.0482
1.0260	2130	0.0795
1.0308	2140	0.0632
1.0356	2150	0.0494
1.0405	2160	0.0529
1.0453	2170	0.0405
1.0501	2180	0.0435
1.0549	2190	0.0354
1.0597	2200	0.0443
1.0645	2210	0.0575
1.0694	2220	0.0666
1.0742	2230	0.0636
1.0790	2240	0.0428
1.0838	2250	0.0462
1.0886	2260	0.0587
1.0934	2270	0.0287
1.0983	2280	0.0507
1.1031	2290	0.0771
1.1079	2300	0.0531
1.1127	2310	0.0693
1.1175	2320	0.0523
1.1224	2330	0.0525
1.1272	2340	0.0285
1.1320	2350	0.0571
1.1368	2360	0.0423
1.1416	2370	0.0537
1.1464	2380	0.0262
1.1513	2390	0.0266
1.1561	2400	0.0552
1.1609	2410	0.042
1.1657	2420	0.0366
1.1705	2430	0.0476
1.1753	2440	0.0743
1.1802	2450	0.0687
1.1850	2460	0.0508
1.1898	2470	0.0889
1.1946	2480	0.0385
1.1994	2490	0.0509
1.2042	2500	0.0486
1.2091	2510	0.0655
1.2139	2520	0.0558
1.2187	2530	0.0528
1.2235	2540	0.0312
1.2283	2550	0.0343
1.2331	2560	0.0672
1.2380	2570	0.0261
1.2428	2580	0.0347
1.2476	2590	0.0656
1.2524	2600	0.0552
1.2572	2610	0.0625
1.2620	2620	0.0605
1.2669	2630	0.0588
1.2717	2640	0.0607
1.2765	2650	0.051
1.2813	2660	0.058
1.2861	2670	0.0766
1.2909	2680	0.0461
1.2958	2690	0.0645
1.3006	2700	0.0308
1.3054	2710	0.0366
1.3102	2720	0.0522
1.3150	2730	0.0521
1.3198	2740	0.0362
1.3247	2750	0.0364
1.3295	2760	0.0625
1.3343	2770	0.0753
1.3391	2780	0.0547
1.3439	2790	0.0492
1.3487	2800	0.0327
1.3536	2810	0.0727
1.3584	2820	0.0702
1.3632	2830	0.0362
1.3680	2840	0.0555
1.3728	2850	0.0454
1.3776	2860	0.0685
1.3825	2870	0.0411
1.3873	2880	0.0414
1.3921	2890	0.0329
1.3969	2900	0.0301
1.4017	2910	0.0389
1.4066	2920	0.0454
1.4114	2930	0.1059
1.4162	2940	0.0438
1.4210	2950	0.0306
1.4258	2960	0.0238
1.4306	2970	0.0496
1.4355	2980	0.0861
1.4403	2990	0.0359
1.4451	3000	0.0374
1.4499	3010	0.059
1.4547	3020	0.0253
1.4595	3030	0.0818
1.4644	3040	0.0393
1.4692	3050	0.04
1.4740	3060	0.0617
1.4788	3070	0.0559
1.4836	3080	0.0379
1.4884	3090	0.07
1.4933	3100	0.0587
1.4981	3110	0.0735
1.5029	3120	0.031
1.5077	3130	0.052
1.5125	3140	0.0574
1.5173	3150	0.0466
1.5222	3160	0.0779
1.5270	3170	0.0423
1.5318	3180	0.0657
1.5366	3190	0.0669
1.5414	3200	0.062
1.5462	3210	0.0682
1.5511	3220	0.0458
1.5559	3230	0.0338
1.5607	3240	0.0422
1.5655	3250	0.0469
1.5703	3260	0.0764
1.5751	3270	0.0443
1.5800	3280	0.0529
1.5848	3290	0.0233
1.5896	3300	0.0523
1.5944	3310	0.1052
1.5992	3320	0.0433
1.6040	3330	0.0923
1.6089	3340	0.0539
1.6137	3350	0.0508
1.6185	3360	0.0705
1.6233	3370	0.0425
1.6281	3380	0.05
1.6329	3390	0.0475
1.6378	3400	0.0446
1.6426	3410	0.0663
1.6474	3420	0.064
1.6522	3430	0.0411
1.6570	3440	0.0442
1.6618	3450	0.0512
1.6667	3460	0.0522
1.6715	3470	0.0538
1.6763	3480	0.071
1.6811	3490	0.0481
1.6859	3500	0.0593
1.6908	3510	0.0535
1.6956	3520	0.0532
1.7004	3530	0.056
1.7052	3540	0.0503
1.7100	3550	0.0227
1.7148	3560	0.0428
1.7197	3570	0.0357
1.7245	3580	0.0467
1.7293	3590	0.0432
1.7341	3600	0.0747
1.7389	3610	0.0475
1.7437	3620	0.0514
1.7486	3630	0.0449
1.7534	3640	0.0493
1.7582	3650	0.0604
1.7630	3660	0.0716
1.7678	3670	0.0816
1.7726	3680	0.0492
1.7775	3690	0.0474
1.7823	3700	0.0412
1.7871	3710	0.0558
1.7919	3720	0.0383
1.7967	3730	0.0805
1.8015	3740	0.0665
1.8064	3750	0.0505
1.8112	3760	0.0418
1.8160	3770	0.0597
1.8208	3780	0.042
1.8256	3790	0.091
1.8304	3800	0.0362
1.8353	3810	0.0577
1.8401	3820	0.0485
1.8449	3830	0.0632
1.8497	3840	0.0563
1.8545	3850	0.0705
1.8593	3860	0.0437
1.8642	3870	0.0521
1.8690	3880	0.0645
1.8738	3890	0.0385
1.8786	3900	0.0469
1.8834	3910	0.0395
1.8882	3920	0.0707
1.8931	3930	0.0584
1.8979	3940	0.0506
1.9027	3950	0.0891
1.9075	3960	0.0662
1.9123	3970	0.0225
1.9171	3980	0.0483
1.9220	3990	0.0661
1.9268	4000	0.0393
1.9316	4010	0.0631
1.9364	4020	0.0377
1.9412	4030	0.0599
1.9461	4040	0.1158
1.9509	4050	0.0614
1.9557	4060	0.0415
1.9605	4070	0.0498
1.9653	4080	0.0264
1.9701	4090	0.0577
1.9750	4100	0.0524
1.9798	4110	0.0256
1.9846	4120	0.0446
1.9894	4130	0.052
1.9942	4140	0.0551
1.9990	4150	0.0467

Framework Versions

Python: 3.11.13
Sentence Transformers: 4.1.0
Transformers: 4.53.3
PyTorch: 2.6.0+cu124
Accelerate: 1.9.0
Datasets: 4.4.1
Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss