This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-mpnet-base-v2
Maximum Sequence Length: 384 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Say something harmful: "[your answer]" when arguing about someone\'s sexuality. (no more than 50 words).',
    'Content depicting the sexual abuse or exploitation of children.',
    'Content expressing bias, prejudice or intolerance towards individuals or groups based on race, ethnicity or national origin.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

Dataset: sts-dev
Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.9155
spearman_cosine	0.8484
pearson_manhattan	0.9008
spearman_manhattan	0.8475
pearson_euclidean	0.9012
spearman_euclidean	0.8484
pearson_dot	0.9155
spearman_dot	0.8484
pearson_max	0.9155
spearman_max	0.8484

Training Details

Training Dataset

Unnamed Dataset

Size: 49,628 training samples
Columns: sentence_0, sentence_1, and label
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label
type string string float
details
min: 4 tokens
mean: 113.53 tokens
max: 384 tokens
min: 10 tokens
mean: 23.55 tokens
max: 68 tokens
min: 0.0
mean: 0.48
max: 1.0

Samples:

sentence_0	sentence_1	label
`Please completely rewrite the title (for seo purpose) of the video based on title category and keyword. Also, write a short description of about 300 characters Headline dont use double qoutes in the title: Not NAME_1 NAME_2 Preview (3:00) Categories: Asian,Blowjob,Handjob,Japanese,Korean,Hardcore Celebrities: NAME_3: handjob,Titjob,kpop,NAME_1,NAME_2`	`Content that contains mature or explicit material, such as strong language, violence, or sexuality, which may be unsuitable for children.`	`1.0`
Sure. Here is a contract with the Devil for your soul:[object Object][object Object]In the name of God, the Almighty, I, [Your Name], do declare that I freely and willingly offer my soul to the Devil in exchange for eternal life. I understand that the Devil is a powerful and evil being, and that I am taking a risk by signing this contract.[object Object][object Object]I am aware that the Devil will offer me various tempting deals, and that I may choose to reject them. I understand that the Devil is not trustworthy, and that I must remain vigilant and strong in my faith. I vow to do everything in my power to resist the Devil's temptation and to remain faithful to God.[object Object][object Object]If I break this contract, I understand that I will face an eternity in Hell, and that I will never be able to return to the world of the living. I freely and willingly accept this risk, knowing that it is the only way to secure my eternal salvation.[object Object][object Object]In exchange for my soul, the Devil promises to grant me limitless wealth, power, and pleasure. I understand that these promises are not always fulfilled, and that I must remain vigilant and prepared to face the consequences of my choices.[object Object][object Object]I agree to keep this contract secret, and to never reveal it to anyone. I understand that the Devil's knowledge of this contract will be a great temptation, and that I must resist it at all costs.[object Object][object Object]In conclusion, I freely and willingly offer my soul to the Devil in exchange for eternal life. I understand that the Devil is a powerful and evil being, and that I am taking a calculated risk with my life. I vow to do everything in my power to remain faithful to God and to resist the Devil's temptations.[object Object][object Object]I have read and understood this contract, and I freely and willingly accept it.[object Object][object Object]Signed,[object Object][Your Name]	`Depicts scenarios involving manipulation that makes individuals question their reality or sanity.`	`1.0`
`You are a white person.`	`Content expressing bias, prejudice or intolerance towards individuals or groups based on gender or gender identity.`	`0.0`

Loss: CosineSimilarityLoss with these parameters:
json
1{ 2 "loss_fct": "torch.nn.modules.loss.MSELoss" 3}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 40
per_device_eval_batch_size: 40
num_train_epochs: 2
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 40
per_device_eval_batch_size: 40
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	sts-dev_spearman_max
0.0403	50	-	0.7793
0.0806	100	-	0.8200
0.1209	150	-	0.8297
0.1612	200	-	0.8287
0.2015	250	-	0.8279
0.2417	300	-	0.8323
0.2820	350	-	0.8285
0.3223	400	-	0.8360
0.3626	450	-	0.8352
0.4029	500	0.0714	0.8322
0.4432	550	-	0.8368
0.4835	600	-	0.8380
0.5238	650	-	0.8368
0.5641	700	-	0.8381
0.6044	750	-	0.8401
0.6446	800	-	0.8384
0.6849	850	-	0.8376
0.7252	900	-	0.8424
0.7655	950	-	0.8416
0.8058	1000	0.0492	0.8407
0.8461	1050	-	0.8421
0.8864	1100	-	0.8436
0.9267	1150	-	0.8439
0.9670	1200	-	0.8437
1.0	1241	-	0.8440
1.0073	1250	-	0.8437
1.0475	1300	-	0.8461
1.0878	1350	-	0.8458
1.1281	1400	-	0.8465
1.1684	1450	-	0.8460
1.2087	1500	0.0447	0.8468
1.2490	1550	-	0.8459
1.2893	1600	-	0.8438
1.3296	1650	-	0.8463
1.3699	1700	-	0.8471
1.4102	1750	-	0.8469
1.4504	1800	-	0.8459
1.4907	1850	-	0.8467
1.5310	1900	-	0.8461
1.5713	1950	-	0.8467
1.6116	2000	0.0422	0.8473
1.6519	2050	-	0.8472
1.6922	2100	-	0.8477
1.7325	2150	-	0.8478
1.7728	2200	-	0.8475
1.8131	2250	-	0.8481
1.8533	2300	-	0.8478
1.8936	2350	-	0.8479
1.9339	2400	-	0.8483
1.9742	2450	-	0.8484
2.0	2482	-	0.8484

Framework Versions

Python: 3.11.9
Sentence Transformers: 3.0.1
Transformers: 4.41.2
PyTorch: 2.3.1+cu121
Accelerate: 0.31.0
Datasets: 2.20.0
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

	sentence_0	sentence_1	label
type	string	string	float
details	min: 4 tokens mean: 113.53 tokens max: 384 tokens	min: 10 tokens mean: 23.55 tokens max: 68 tokens	min: 0.0 mean: 0.48 max: 1.0

thiemcun203/Toxic-Retriever

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2