You can deploy this model and start earning money today!

Netta1994/
setfit_baai_newrelic_gpt-4o_improved-cot_chat_few_shot_only_reasoning_1726751494.1082

Name: setfit_baai_newrelic_gpt-4o_improved-cot_chat_few_shot_only_reasoning_1726751494.1082
Availability: InStock

setfit

safetensors

bert

sentence-transformers

text-classification

generated_from_setfit_trainer

2209.11055

BAAI/bge-base-en-v1.5

finetune

model-index

Views

No views yet

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label

Examples

"Reasoning:\nThe answer provided does not align with the content of the documents. It offers general advice on saving money rather than specific insights from the provided documents that relate to ORGANIZATION's specific guidelines or context about financial prudence or savings.\n\nEvaluation: Bad"
'Reasoning:\nThe answer is correct; it properly identifies several specific pet peeves mentioned in the document, such as sabotaging work, unwanted advances, and derogatory comments. However, the answer contains numerous repetitions of names and accidental insertions of text fragments which make it difficult to read. This detracts from the clarity and quality, despite being factually correct.\n\nEvaluation: Bad'
"Reasoning:\nThe answer given does not provide any information or instructions about accessing the company's training resources. Instead, it lists various unrelated methods such as accessing personal documents, managing passwords, and requesting learning budgets, based on the provided documents. The answer does not directly address the question.\n\nEvaluation: Bad"

'Reasoning:\nThe answer accurately captures the key points from the document regarding how feedback should be given. It mentions giving feedback at the time of the event, focusing on the situation rather than the person, aiming to help rather than shame, being clear and direct, and showing appreciation. It also covers tips for receiving feedback. The answer presents these points clearly and is aligned with the provided document.\n\nEvaluation: Good'
'Reasoning:\nThe answer effectively captures the reasons for proactively sharing information from high-level meetings, such as providing transparency, ensuring that team members have the necessary context, aligning the team, and fostering a sense of purpose. These points are supported by the provided documents, particularly Document 4.\n\nEvaluation: Good'
'Reasoning:\nThe provided answer accurately describes the procedure for reporting car travel expenses for reimbursement, including precise details such as tracking kilometers and sending details to specific email addresses. This information directly corresponds to the content provided in Document 1.\n\nFinal result: Good'

Evaluation

Metrics

Label	Accuracy
all	0.5522

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

python

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot_chat_few_shot_only_reasoning_1726751494.1082")
# Run inference
preds = model("Reasoning:
The answer given in the response correctly reflects the information in Document 1, which states that questions regarding travel reimbursement should be directed to finance@ORGANIZATION_2.<89312988>. The required email address is present in the document and clearly mentions who to contact.
Evaluation: Good")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	21	47.4462	85

Label	Training Sample Count
0	32
1	33

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0061	1	0.2243	-
0.3067	50	0.2608	-
0.6135	100	0.2456	-
0.9202	150	0.1701	-
1.2270	200	0.0069	-
1.5337	250	0.0026	-
1.8405	300	0.0021	-
2.1472	350	0.002	-
2.4540	400	0.0018	-
2.7607	450	0.0016	-
3.0675	500	0.0015	-
3.3742	550	0.0015	-
3.6810	600	0.0014	-
3.9877	650	0.0014	-
4.2945	700	0.0014	-
4.6012	750	0.0013	-
4.9080	800	0.0013	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.0
Transformers: 4.44.0
PyTorch: 2.4.1+cu121
Datasets: 2.19.2
Tokenizers: 0.19.1

Citation

BibTeX

bibtex

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}