This model is a fine-tuned version of answerdotai/ModernBERT-base on the None dataset.
It achieves the following results on the evaluation set:
Loss: 0.1729
Mse: 0.1729
Model description
This is a modernbert model with a regression head designed to predict the Content score of a summary.
The input should be the summary + [sep] + source.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("wesleymorris/modernbert-content", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("wesleymorris/modernbert-content")
def get_score(summary: str,
source: str):
text = summary+tokenizer.sep_token+source
inputs = tokenizer(text, return_tensors = 'pt')
return float(model(**inputs).logits[0])
Corpus
It was trained on a corpus of 4,233 summaries of 101 sources compiled by Botarleanu et al. (2022).
The summaries were graded by expert raters on 6 criteria: Details, Main Point, Cohesion, Paraphrasing, Objective Language, and Language Beyond the Text.
A principle component analyis was used to reduce the dimensionality of the outcome variables to two.
Content includes Details, Main Point, Paraphrasing and Cohesion
Contact
This model was developed by LEAR Lab at Vanderbilt University. For questions or comments about this model, please contact wesley.g.morris@vanderbilt.edu.
Intended uses & limitations
This model can be used to predict human scores of content for a summary.
The scores are normalized such that 0 is the mean of the training data and 1 is one standard deviation from the mean.
Training and evaluation data
Before the finetuning step, the model was pretrained on a very large synthetic dataset.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 5e-05
train_batch_size: 8
eval_batch_size: 16
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
