You can deploy this model and start earning money today!

Netta1994/
setfit_baai_newrelic_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_eval

Name: setfit_baai_newrelic_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_eval
Rating: 2.5 (11 reviews)

setfit

safetensors

bert

sentence-transformers

text-classification

generated_from_setfit_trainer

2209.11055

BAAI/bge-base-en-v1.5

finetune

model-index

Views

No views yet

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label

Examples

'Reasoning why the answer may be good:\n1. Context Grounding: The given answer is well-supported by the document provided. It describes the features of the Womens Drew Argo Black Smooth Leather sandal, matching the details from the text.\n2. Relevance: It answers the implicit question about the features of the Women's Drew Argo Black Smooth Leather sandal correctly, as all features listed in the answer are found in the document.\n3. Conciseness: The answer is clear, direct, and to the point, listing the sandal's features without unnecessary information.\n\nReasoning why the answer may be bad:\n1. Context Grounding: The core issue is that the user asked, "How to Clean Patent Leather," but the answer provides features of a specific leather sandal. This does not utilize the document correctly since it does not address cleaning methods at all.\n2. Relevance: The answer does not address the specific question asked. The user inquired about cleaning patent leather, not about the features of a sandal.\n3. Conciseness: While the answer is concise regarding the sandal's features, it fails entirely to be concise about the actual question asked.\n\nFinal Result:'
'Reasoning why the answer may be good:\n1. Relevance: The answer is focused on identifying the name "Al-Rassan" in relation to the IBM Thinkpad, which would be correct if the question were about the name of the Thinkpad mentioned in the document.\n2. Context Grounding: The document supports the identification of "Al-Rassan" as the name associated with an IBM Thinkpad.\n3. Conciseness: The answer is short and to the point.\n\nReasoning why the answer may be bad:\n1. Relevance: The answer does not address the actual question, which is about the meaning of "Madigan" in Irish.\n2. Context Grounding: The provided document does not contain any information on the meaning of "Madigan" in Irish, which means the answer is not grounded in the context of the document.\n3. Conciseness: Although the answer is concise, it is not relevant to the question asked.\n\nFinal result:'
'Reasoning:\n\nGood Aspects: \n1. Context Grounding: The answer mentions moving to the editor, which is grounded in the provided document where it states, "To add a calendar you need to move to the Editor."\n2. Relevance: The answer attempts to address the question related to setting up a calendar for call back requests.\n\nBad Aspects: \n1. Context Grounding: The document explicitly mentions that does not have a calendar feature to book services or appointments, which contradicts the instructions provided in the answer. This makes the answer potentially misleading.\n2. Conciseness: The phrase "following the steps provided in this article" is vague and unhelpful since no specific steps are provided in the document snippet.\n3. Correct Instructions: The answer doesn’t provide detailed instructions or any steps on how to set up a calendar, as the document states that this feature doesn’t exist and users areencouraged to vote for it.\n\nFinal Result:'

'Reasoning why the answer may be good:\n1. Context Grounding: The answer lists the portals discussed in the document. The answer names IM Portal, Simple Portal, Stargate Portal, Integra Mod3, and Board3 Portal, which are all mentioned in the provided text.\n2. Relevance: The answer directly addresses the question by listing the portal options without going into unrelated topics or additional commentary.\n3. Conciseness: The answer is brief and to the point, containing only the relevant information required by the question.\n\nReasoning why the answer may be bad:\n1. Context Grounding: The elegance of context inclusion could be debated, but all named portals are mentioned within the document. \n2. Relevance: The answer slightly deviates from the question by not keeping Integra Mod3 and Board3 Portal names precisely as discussed but overall maintains relevance.\n3. Conciseness: It is concise and doesn’t include any extraneous information.\n\nFinal result: ****'
'Reasoning why the answer may be good:\n1. Context Grounding: The answer leverages specific strategies that are mentioned in the provided document, such as hiding in a laundry basket and picking a hiding place after the seeker checks a room.\n2. Relevance: The strategies suggested are directly related to becoming a master at hide and seek, which is exactly what the question asks.\n3. Conciseness: The answer is relatively focused and includes several practical tips without excessive elaboration.\n\nReasoning why the answer may be bad:\n1. Context Grounding: It misses other valuable ideas from the document like using long edges, curtains, yard hiding spots, and decoys, which could provide a much more comprehensive answer.\n2. Relevance: While the answer is relevant, it introduces examples not as prominently detailed in the document, like hiding in plain sight behind multi-colored areas.\n3. Conciseness: Some elements could be interpreted as slightly redundant or not entirely necessary, such as the additional example of standing out behind a red couch.\n\nFinal result: '
"### Reasoning:\n\n#### Why the answer may be good:\n1. Context Grounding: The answer aligns well with the content of the provided document, which emphasizes the importance of Open Data over Open Source in resolving issues of vendor lock-in and data migration.\n2. Relevance: The answer specifically addresses the proposed solution, which is indeed to place more emphasis on Open Data formats.\n3. Conciseness: The answer is to the point and avoids delving into extraneous details.\n\n#### Why the answer may be bad:\n1. Context Grounding: There's a minor omission in the answer regarding the methods suggested to achieve the solution (convincing or forcing websites). It should be explicitly included for complete context alignment.\n2. Conciseness: The explanation around standardizing on Open Data formats could be more succinct.\n\n### Final Result:\nDespite the minor omission, the answer sufficiently captures the key solution proposed in the document and maintains relevance and conciseness.\n\n"

Evaluation

Metrics

Label	Accuracy
all	0.6567

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

python

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_eval")
# Run inference
preds = model("Reasoning why the answer may be good:
1. **Context Grounding**: The answer is directly supported by the information in the provided document, which indicates that queries regarding travel reimbursements should be directed to the finance department.
2. **Relevance**: The answer correctly identifies the appropriate contact for travel reimbursement inquiries.
3. **Conciseness**: The answer is short and addresses the question directly without unnecessary information.
4. **Specificity**: The answer provides a specific email address for contacting finance, aligning with the context provided in the document.

Reasoning why the answer may be bad:
- There is no reasoning suggesting the given answer is bad based on the provided criteria and document context.

Final Result:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	45	129.3891	274

Label	Training Sample Count
0	314
1	326

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0006	1	0.2574	-
0.0312	50	0.2665	-
0.0625	100	0.2506	-
0.0938	150	0.2524	-
0.125	200	0.2531	-
0.1562	250	0.254	-
0.1875	300	0.2536	-
0.2188	350	0.2504	-
0.25	400	0.2404	-
0.2812	450	0.1953	-
0.3125	500	0.1447	-
0.3438	550	0.0954	-
0.375	600	0.0641	-
0.4062	650	0.0549	-
0.4375	700	0.0426	-
0.4688	750	0.0306	-
0.5	800	0.0347	-
0.5312	850	0.0119	-
0.5625	900	0.0123	-
0.5938	950	0.0158	-
0.625	1000	0.0117	-
0.6562	1050	0.015	-
0.6875	1100	0.0185	-
0.7188	1150	0.0129	-
0.75	1200	0.0084	-
0.7812	1250	0.0057	-
0.8125	1300	0.0084	-
0.8438	1350	0.0113	-
0.875	1400	0.0036	-
0.9062	1450	0.004	-
0.9375	1500	0.0053	-
0.9688	1550	0.0072	-
1.0	1600	0.0041	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.1
Transformers: 4.44.0
PyTorch: 2.4.0+cu121
Datasets: 3.0.0
Tokenizers: 0.19.1

Citation

BibTeX

bibtex

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}