You can deploy this model and start earning money today!

zks2856/
PairQual-Scorer-en

Name: PairQual-Scorer-en
Availability: InStock

safetensors

bert

BAAI/bge-base-en-v1.5

finetune

mit

Views

No views yet

PairQual-Scorer-en

Model summary

This evaluator is designed to assess the overall quality of English corpora. It was trained on 500k annotations generated by GPT-4o-mini, using samples from various sources such as Common Crawl, RedPajama, Wikipedia.

How to use in transformers

To load this model, use the following code:

python

from transformers import BertConfig, AutoTokenizer
from modeling_bert_regression import BertForRegression

tokenizer = AutoTokenizer.from_pretrained("zks2856/PairQual-Scorer-en")
config = BertConfig.from_pretrained("zks2856/PairQual-Scorer-en")
model = BertForRegression.from_pretrained("zks2856/PairQual-Scorer-en", config=config, trust_remote_code=True)

text = "This is a test sentence."
inputs = tokenizer(text, return_tensors="pt", padding="longest", truncation=True)
pooled, score = model(**inputs)

print(score)

After training with the pairwise paradigm, the scores mostly range from -8 to 8, where a higher score indicates higher quality. Users are encouraged to check the score distribution of their corpora and determine an appropriate threshold based on their own criteria. This model has only been trained to evaluate the quality of English samples. Using it on other languages may lead to undefined behavior.

Training

The evaluator was trained on 500,000 pairs of English samples along with their quality preferences, generated by GPT-4o-mini. The training followed a pairwise learning paradigm that simultaneously increases the score of the higher-quality sample and decreases that of the lower-quality counterpart.

Below is the prompt used for GPT annotations:

You are a professional evaluator of Chinese text quality. Please assess the quality of the texts based on the following criteria:

Readability and Coherence: Consider whether the text is fluent, easy to understand, free of formatting issues or gibberish, and uses appropriate grammar and vocabulary.
Educational Value: Consider whether the text contains sufficient and accurate scientific knowledge and objective facts, has educational significance for humans, and avoids logical contradictions or false information.
Data Cleanliness: Consider whether the text has been cleaned of noise, off-topic content, personal sensitive information, advertisements, links, copyright notices, etc.

Based on these criteria, evaluate the quality of the following two texts. The texts may have been truncated, so do not let length or order influence your judgment.

Text 0:
{{text0}}

Text 1:
{{text1}}

Please output only the index of the higher-quality text. Do not output anything else.

We added a regression head to bge-base-en-v1.5 and trained the model for 3 epochs with a learning rate of 1e-5.

Training Details:

Model: bge-base-en-v1.5 with a regression head
Loss: binary cross-entropy loss
Dataset: 500,000 samples from GPT annotations
Epochs: 3
Learning Rate: 1e-5