SentenceTransformer based on distilbert/distilbert-base-uncased

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Semantic Similarity

• Datasets: sts-dev and sts-test
• Evaluated with EmbeddingSimilarityEvaluator

Training Details

Training Dataset

csv

• Dataset: csv
• Size: 4,480 training samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	int
  details	min: 6 tokens mean: 15.14 tokens max: 50 tokens	min: 6 tokens mean: 15.07 tokens max: 52 tokens	0: ~14.20% 1: ~11.60% 2: ~18.40% 3: ~23.30% 4: ~21.70% 5: ~10.80%

• Samples:

  sentence1	sentence2	score
  A man is speaking.	A man is spitting.	1
  Austrian found hoarding 56 stolen skulls in home museum	Austrian man charged after 56 human skulls are found at his home	4
  Mitt Romney wins Republican primary in Indiana	Romney wins Florida Republican primary	2

• Loss: CosineSimilarityLoss with these parameters:
  json

  1{ 2 "loss_fct": "torch.nn.modules.loss.MSELoss" 3}

Evaluation Dataset

csv

• Dataset: csv
• Size: 560 evaluation samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 560 samples:

  	sentence1	sentence2	score
  type	string	string	int
  details	min: 5 tokens mean: 14.41 tokens max: 44 tokens	min: 5 tokens mean: 14.28 tokens max: 42 tokens	0: ~12.86% 1: ~16.96% 2: ~14.82% 3: ~18.21% 4: ~26.43% 5: ~10.71%

• Samples:

  sentence1	sentence2	score
  An airplane is flying in the air.	A South African Airways plane is flying in a blue sky.	3
  A television, upholstered chair, and coffee stable in a bright room.	A leather couch and wooden table in a living room.	2
  Red panda’s short-lived zoo escape	India’s march to Mars	0

• Loss: CosineSimilarityLoss with these parameters:
  json

  1{ 2 "loss_fct": "torch.nn.modules.loss.MSELoss" 3}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 4
• warmup_ratio: 0.1

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 4.1.0
• Transformers: 4.49.0
• PyTorch: 2.3.0.post101
• Accelerate: 1.10.1
• Datasets: 3.3.2
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers