A BERT-based language model further pre-trained from the checkpoint of
SciBERT.
The dataset gathered is a balance between scientific and general works in agriculture domain and encompassing knowledge from different areas of agriculture research and practical knowledge.
The corpus contains 1.2 million paragraphs from National Agricultural Library (NAL) from the US Gov. and 5.3 million paragraphs from books and common literature from the Agriculture Domain.
The self-supervised learning approach of MLM was used to train the model.
1from transformers import pipeline
2fill_mask = pipeline(
3 "fill-mask",
4 model="recobo/agriculture-bert-uncased",
5 tokenizer="recobo/agriculture-bert-uncased"
6)
7fill_mask("[MASK] is the practice of cultivating plants and livestock.")
