Example

python

from transformers import pipeline

classifier = pipeline("text-classification", model="agentlans/bge-small-en-v1.5-ultrafineweb-vs-pile-classifier")
classifier("Your text here.")

Limitations

Tends to be overly strict, labelling most texts outside training data as low quality
English only
May be biased against some text types such as source code and personal blogs

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Combined Score	Input Tokens Seen
0.2893	1.0	19958	0.2926	0.9061	2.1448	20436992
0.2397	2.0	39916	0.3127	0.9076	2.1194	40873984
0.2	3.0	59874	0.3279	0.9109	2.0605	61310976
0.1576	4.0	79832	0.3887	0.9080	2.1119	81747968
0.1127	5.0	99790	0.4688	0.9069	2.1308	102184960

Framework versions

Transformers 4.51.3
Pytorch 2.6.0+cu124
Datasets 3.2.0
Tokenizers 0.21.0