SentenceTransformer

Model Details

Model Description

• Model Type: Sentence Transformer

• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (transformer): Transformer(
    (auto_model): XLMRobertaLoRA(
      (roberta): XLMRobertaModel(
        (embeddings): XLMRobertaEmbeddings(
          (word_embeddings): ParametrizedEmbedding(
            250002, 1024, padding_idx=1
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
          (token_type_embeddings): ParametrizedEmbedding(
            1, 1024
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
        )
        (emb_drop): Dropout(p=0.1, inplace=False)
        (emb_ln): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (encoder): XLMRobertaEncoder(
          (layers): ModuleList(
            (0-23): 24 x Block(
              (mixer): MHA(
                (rotary_emb): RotaryEmbedding()
                (Wqkv): ParametrizedLinearResidual(
                  in_features=1024, out_features=3072, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
                (inner_attn): FlashSelfAttention(
                  (drop): Dropout(p=0.1, inplace=False)
                )
                (inner_cross_attn): FlashCrossAttention(
                  (drop): Dropout(p=0.1, inplace=False)
                )
                (out_proj): ParametrizedLinear(
                  in_features=1024, out_features=1024, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
              )
              (dropout1): Dropout(p=0.1, inplace=False)
              (drop_path1): StochasticDepth(p=0.0, mode=row)
              (norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
              (mlp): Mlp(
                (fc1): ParametrizedLinear(
                  in_features=1024, out_features=4096, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
                (fc2): ParametrizedLinear(
                  in_features=4096, out_features=1024, bias=True
                  (parametrizations): ModuleDict(
                    (weight): ParametrizationList(
                      (0): LoRAParametrization()
                    )
                  )
                )
              )
              (dropout2): Dropout(p=0.1, inplace=False)
              (drop_path2): StochasticDepth(p=0.0, mode=row)
              (norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
            )
          )
        )
        (pooler): XLMRobertaPooler(
          (dense): ParametrizedLinear(
            in_features=1024, out_features=1024, bias=True
            (parametrizations): ModuleDict(
              (weight): ParametrizationList(
                (0): LoRAParametrization()
              )
            )
          )
          (activation): Tanh()
        )
      )
    )
  )
  (pooler): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (normalizer): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

Training Details

Training Dataset

Unnamed Dataset

• Size: 584,355 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 6 tokens mean: 17.41 tokens max: 42 tokens	min: 5 tokens mean: 119.19 tokens max: 1979 tokens

• Samples:

  anchor	positive
  What resources and tools are recommended for busy brides planning their weddings	If you are planning on spending a little bit of time on your wedding planning, here is part 2 of my series of great resources and tools for wedding planning that every busy bride should know about. The previous instalment can be viewed here.
  How many girls were raised in the house described	This house is where my parents proudly hung up our diplomas. This house is where 3 girls were raised.
  Where did the narrator's dad always barbecue for Easter	This house is where my dad always barbequed for Easter, rain or shine. This house is where we welcomed family and friends on their first visit to the United States.

• Loss: cachedselfloss2.CachedInfonce with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Evaluation Dataset

Unnamed Dataset

• Size: 18,073 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 6 tokens mean: 17.52 tokens max: 40 tokens	min: 5 tokens mean: 107.58 tokens max: 1832 tokens

• Samples:

  anchor	positive
  What significant role did the character Raven portray in early cinema	He played cynical tough guys in modern films, but then branched into westerns where for the most part he was the gallant hero. In fact the ultimate gallant white knight hero in Shane. His part as Raven is a difficult one, yet he pulls it off. He's a cold blooded contract killer, one of the earliest ever portrayed as a film protagonist. Yet he's human and you see flashes of it, his concern for cats. As a cat lover, I can sure identify with that. Raven is also one of the earliest characters in cinema who talks about child abuse making him what he is. Groundbreaking when you think about it. Next to Ladd, the biggest kudos have to go to Laird Cregar, borrowed from 20th Century Fox to play Willard Gates. Gates is a top company executive with Marshall's firm which is a defense contractor which is why the Senate is interested in him. He's basically a jerk who thinks he's so clever. Veronica Lake gets to him real easy because of his weakness for the nightclub scene.
  What are the key features and characteristics of the Majestic Pure Dead Sea Mud Mask	At the same time, it can be used all over your body, not just face. This way, you can clear any part of your skin from its impurities. An additional feature that caught our eyes immediately was the beautifully designed packaging that makes this affordable product look high-end. The combination of grey, black, and blue colors will easily make it stand out in any beauty shop. - Good for sensitive and dry skin[object Object]- Affordable price[object Object]- Treats many different skin conditions[object Object]- Not good for oily skin[object Object]- Can feel a bit oily[object Object]Majestic Pure Dead Sea Mud Mask Review[object Object]Majestic Pure is a well-known brand among the beauty and skincare community. They make affordable, natural products and are oftentimes among the celebrity favorites.
  What benefits does this product provide for the skin	This will provide you with softer skin that glows. At the same time, it will help you deal with your clogged pores and provide you the necessary daily acne treatment.

• Loss: cachedselfloss2.CachedInfonce with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 800
• per_device_eval_batch_size: 800
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.4.1
• Transformers: 4.49.0
• PyTorch: 2.3.1+cu121
• Accelerate: 1.5.2
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

CachedInfonce