You can deploy this model and start earning money today!

mdh266/
arxivist

Name: arxivist
Availability: InStock

generated_from_trainer

google-bert/bert-base-uncased

Views

No views yet

arxivist

This model is a fine-tuned version of google-bert/bert-base-uncased on data collected from arxiv. See the blog post on this model for more information. This was used for my own educational purposes.

It achieves the following results on the evaluation set:

Loss: 0.4750
Roc Auc: 0.9709

Model description

The model is a BertForSequenceClassification model fine-tuned using the Hugging Face transformers library. The base model used was google-bert/bert-base-uncased.

Intended uses & limitations

This model is fine tuned to predict paper abstracts into either "Artificial Intelligence", "Information Retrieval" and "Robotics".

Training and evaluation data

See the blog post on this model for more information.

Training procedure

The model was trained for 5 epochs with a learning rate of 1e-4 and a batch size of 16 for training and 8 for evaluation. Dynamic padding was used during training.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 16
eval_batch_size: 8
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Roc Auc
0.461	1.0	90	0.6019	0.9568
0.2395	2.0	180	0.2899	0.9775
0.1343	3.0	270	0.3588	0.9808
0.0481	4.0	360	0.4495	0.9771
0.0264	5.0	450	0.4750	0.9709

Framework versions

Transformers 4.55.0
Pytorch 2.6.0+cu124
Datasets 4.0.0
Tokenizers 0.21.4

mdh266/arxivist