T5 Base Model for Named Entity Recognition (NER, CoNLL-2003)
In this repository, we open source a T5 Base model, that was fine-tuned on the official CoNLL-2003 NER dataset.
We use the great
TANL library from Amazon for fine-tuning the model.
The exact approach of fine-tuning is presented in the "TANL: Structured Prediction as Translation between Augmented Natural Languages"
paper from Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang and Stefano Soatto.
Fine-Tuning
We use the same hyper-parameter settings as used in the official implementation with one minor change. Instead of using 8 V100 GPUs, we train the model
on one V100 GPU and used gradient accumulation. The slighly modified configuration file (config.ini) then looks like:
1[conll03]
2datasets = conll03
3model_name_or_path = t5-base
4num_train_epochs = 10
5max_seq_length = 256
6max_seq_length_eval = 512
7per_device_train_batch_size = 4
8per_device_eval_batch_size = 4
9do_train = True
10do_eval = True
11do_predict = True
12gradient_accumulation_steps = 8
It took around 2 hours to fine-tune that model on the 14,041 training sentences of CoNLL-2003 dataset.
Evaluation
On the development set, the following evaluation results could be achieved:
1{
2"entity_precision": 0.9536446086664427,
3"entity_recall": 0.9555705149781218,
4"entity_f1": 0.9546065904505716,
5"entity_precision_no_type": 0.9773261672824992,
6"entity_recall_no_type": 0.9792998990238977,
7"entity_f1_no_type": 0.9783120376597176
8}
The evaluation results on the test set looks like:
1{
2"entity_precision": 0.912182296231376,
3"entity_recall": 0.9213881019830028,
4"entity_f1": 0.9167620893155995,
5"entity_precision_no_type": 0.953900087642419,
6"entity_recall_no_type": 0.9635269121813032,
7"entity_f1_no_type": 0.9586893332158901
8}
To summarize: On the development set, 95.46% F1-Score and 91.68% on test set were achieved with this model. The paper reported a F1-Score of 91.7%.
License
The models is licensed under
MIT.
Acknowledgments
Thanks to the generous support from the
Hugging Face team,
it is possible to download both cased and uncased models from their S3 storage 🤗
