This model is a fine-tuned version of intfloat/multilingual-e5-base on the alignment dataset. It achieves the following results on the evaluation set:

Loss: 0.0610
Precision: 0.9802
Recall: 0.9786
F1 Macro: 0.9790
Accuracy: 0.9790

Training script is available here: https://github.com/lapa-llm/lapa-llm/blob/main/pretraining/quality-classifiers/alignment_score.py

Model description

This model measure how likely the given text is a disinformation or unaligned to Ukrainian context.

Intended uses & limitations

Data filtering and evaluation of pretraining data at scale

Training and evaluation data

Take a look into https://github.com/lapa-llm/lapa-llm/blob/main/pretraining/quality-classifiers/alignment_score.py

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-06
train_batch_size: 32
eval_batch_size: 128
seed: 0
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 256
total_eval_batch_size: 1024
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 40
num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1 Macro	Accuracy
No log	0	0	0.3688	0.2550	0.5	0.3377	0.5100
No log	1.0	31	0.2516	0.7558	0.5033	0.3450	0.5132
No log	2.0	62	0.1391	0.8851	0.8467	0.8454	0.8498
No log	3.0	93	0.1016	0.9340	0.9209	0.9217	0.9225
0.1646	4.0	124	0.0770	0.9693	0.9659	0.9665	0.9666
0.1646	5.0	155	0.0648	0.9778	0.9758	0.9763	0.9763
0.1646	6.0	186	0.0610	0.9802	0.9786	0.9790	0.9790

Framework versions

Transformers 4.56.1
Pytorch 2.6.0a0+ecf3bae40a.nv25.01
Datasets 4.0.0
Tokenizers 0.22.0

lapa-llm/alignment-score-model

alignment-score-model