SentenceTransformer based on allenai/scibert_scivocab_uncased

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: allenai/scibert_scivocab_uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Triplet

• Dataset: val-eval
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 19,392 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 5 tokens mean: 102.27 tokens max: 512 tokens	min: 3 tokens mean: 356.76 tokens max: 512 tokens	min: 3 tokens mean: 326.25 tokens max: 512 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  Let $f:\R^D
  ightarrow\R$ be an $L$-hidden layer multi-layer perceptron (MLP) such that [ f(xv)=\sigma_{L+1}ig(\wv^ op\sigma_L(\Wm_L\sigma_{L-1}(\Wm_{L-1}\dots\sigma_1(\Wm_1xv)))ig), ] with $\wv\in\R^{M}$, $\Wm_1\in\R^{M imes D}$ and $\Wm_\ell\in\R^{M imes M}$ for $\ell=2,\dots, L$, and $\sigma_i$ for $i=1,\dots,L+1$ is an entry-wise activation function. For any MLP $f$ and a classification threshold $ au$ let $C_{f, au}$ be a binary classifier that outputs YES for a given input $xv$ if $f(xv) \leq au$ and NO otherwise.  space{3mm} Assume $\sigma_{L+1}$ is the element-wise extbf{sigmoid} function and $C_{f,  rac{1}{2}}$ is able to obtain a high accuracy on a given binary classification task $T$. Let $g$ be the MLP obtained by multiplying the parameters extbf{in the last layer} of $f$, i.e. $\wv$, by 2. Moreover, let $h$ be the MLP obtained by replacing $\sigma_{L+1}$ with elemen...	To analyze the options regarding the accuracy of the classifiers $C_{g, \frac{1}{2}}$, $C_{h, 0}$, and $C_{q, 0}$ compared to $C_{f, \frac{1}{2}}$, let’s go through each choice:[object Object][object Object]1. [object Object][object Object][object Object] This statement is [object Object]. The MLP $g$ is obtained by multiplying the weights of the last layer ($\wv$) by 2. Since the sigmoid activation function is monotonic, this transformation does not change the sign of the output, only scales it. The threshold of $\frac{1}{2}$ remains valid for classification because if $f(xv) > \frac{1}{2}$, then $g(xv) = 2f(xv) > 1$, which still results in a "YES" classification. Similarly, if $f(xv) < \frac{1}{2}$, then $g(xv) < 1$, leading to a "NO" classification. Therefore, the accuracy of $C_{g, \frac{1}{2}}$ should be the same as that of $C_{f, \frac{1}{2}}$.[object Object][object Object]2. [object Object][object Object][object Object]...	To evaluate the accuracy of the classifiers $C_{g, \frac{1}{2}}$, $C_{h, 0}$, and $C_{q, 0}$ in comparison to $C_{f, \frac{1}{2}}$, let's consider the transformations applied to the original multi-layer perceptron (MLP) $f$:[object Object][object Object]1. [object Object][object Object][object Object] This option raises a concern about the transformation applied to the last layer of the MLP. While it is true that multiplying the weights by 2 could affect the scale of the outputs, it does not necessarily imply a decrease in accuracy. The classification decision depends on the threshold and the relative position of the outputs. Hence, one could argue that the accuracy of $C_{g, \frac{1}{2}}$ might not diverge significantly from that of $C_{f, \frac{1}{2}}$, but the relationship is not straightforward without further analysis.[object Object][object Object]2. [object Object][object Object][object Object] Here, the concern is a...
  What is the complexity of concatenation of two conc-trees with heights $h_1$ and[object Object]$h_2$?	To analyze the complexity of concatenating two conc-trees with heights ( h_1 ) and ( h_2 ), we can refer to the provided concatenation algorithm. The algorithm performs concatenation based on the height difference between the two trees, defined as [object Object].[object Object][object Object]1. [object Object]: If the height difference, [object Object], is within the range ([-1, 1]), the algorithm creates a new root node that combines the two trees directly. This operation requires constant time, ( O(1) ).[object Object][object Object]2. [object Object]: If the height difference exceeds 1, the algorithm executes recursive calls to balance the trees:[object Object] - If [object Object] is shorter than [object Object], it recursively concatenates the right subtree of [object Object] with [object Object] and adjusts the structure accordingly.[object Object] - The recursion aims to reduce the height difference between the two trees, effectively addressing the taller tree until the heights are balanced.[object Object][object Object]3. [object Object]: Each recursive call focuses on reducing the h...	The complexity of concatenating two concatenation trees (or conc-trees) with heights ( h_1 ) and ( h_2 ) can be understood pretty easily. When you concatenate two trees, you essentially combine them into a single tree. [object Object][object Object]Let's think about it this way: the height of the resulting tree would be determined by the taller of the two trees, plus one for the new root that connects them. So, the height of the new tree would be ( \max(h_1, h_2) + 1 ).[object Object][object Object]Now, when it comes to the complexity of this operation, we can say it takes time proportional to the heights of the trees because you may need to traverse one of the trees to attach it to the other. This means the time complexity for concatenating these two trees is ( O(h_1 + h_2) ). [object Object][object Object]So, in summary, concatenating two conc-trees of heights ( h_1 ) and ( h_2 ) has a complexity of ( O(h_1 + h_2) ). This is pretty straightforward since you just combine them and adjust the height!
  Suppose you have a search engine that retrieves the top 100 documents and[object Object]achieves 90% precision and 20% recall. You modify the search engine to[object Object]retrieve the top 200 and mysteriously, the precision stays the same. Which one[object Object]is CORRECT?	To address the question, we need to analyze the implications of the changes made to the search engine regarding precision and recall, two critical metrics used in evaluating information retrieval systems.[object Object][object Object]### Definitions:[object Object]- [object Object] is defined as the ratio of relevant documents retrieved to the total number of documents retrieved. Mathematically, it is expressed as:[object Object] [[object Object] P = \frac{TP}{TP + FP}[object Object] ][object Object] where (TP) is the number of true positives (relevant documents retrieved) and (FP) is the number of false positives (irrelevant documents retrieved).[object Object][object Object]- [object Object] is defined as the ratio of relevant documents retrieved to the total number of relevant documents available. It is given by:[object Object] [[object Object] R = \frac{TP}{TP + FN}[object Object] ][object Object] where (FN) is the number of false negatives (relevant documents not retrieved).[object Object][object Object]### Given Information:[object Object]From the initial state of the search engine:[object Object]- Top 100 documents retrieved[object Object]- Precision = 90% (0.90)[object Object]- Recall = 20% (0.20)[object Object][object Object]Let’s derive some v...	To analyze the multiple-choice question regarding the search engine's precision and recall, we will follow the structured approach outlined in the instructions.[object Object][object Object]### Core Concepts[object Object]1. [object Object]: The ratio of relevant documents retrieved to the total documents retrieved. It is a measure of the accuracy of the search results.[object Object]2. [object Object]: The ratio of relevant documents retrieved to the total relevant documents available. It measures the ability of the search engine to find all relevant documents.[object Object]3. [object Object]: The harmonic mean of precision and recall, providing a single metric to evaluate the performance of a search engine.[object Object][object Object]### Given Information[object Object]- Initial retrieval: Top 100 documents[object Object]- Precision: 90% (0.9)[object Object]- Recall: 20% (0.2)[object Object]- Modified retrieval: Top 200 documents[object Object]- Precision remains at 90%[object Object][object Object]### Analyzing Each Option[object Object][object Object]1. [object Object][object Object] - [object Object]: Incorrect[object Object] - [object Object]: Recall is defined as the number of relevant documents retrieved divided by the ...

• Loss: TripletLoss with these parameters:
  json

  1{ 2 "distance_metric": "TripletDistanceMetric.EUCLIDEAN", 3 "triplet_margin": 5 4}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• num_train_epochs: 1
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 3.4.1
• Transformers: 4.48.2
• PyTorch: 2.5.1+cu124
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

TripletLoss