Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. Recently, several studies have explored the utility and efficacy of contextual models in the clinical, medical, and biomedical domains (BioBERT, ClinicalBERT, SciBERT, BlueBERT However, while there is a growing interest in developing language models for more specific domains, the current trend appears to prefer re-training general-domain models on specialized corpora rather than developing models from the ground up with specialized vocabulary. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. However, in fields requiring specialized terminology, such as pathology, these models often fail to perform adequately. One of the major reasons for this limitation is because BERT employs Word-Pieces for unsupervised input tokenization, a technique that relies on a predetermined set of Word-Pieces. The vocabulary is built such that it contains the most commonly used words or subword units and as a result, any new words can be represented by frequent subwords. Although WordPiece was built to handle suffixes and complex compound words, it often fails with domain-specific terms. For example, while ClinicalBERT successfully tokenizes the word 'endpoint' as ['end', '##point'], it tokenize the word 'carcinoma' as ['car', '##cin', '##oma'] in which the word lost its actual meaning and replaced by some non-relevant junk words, such as `car'. The words which was replaced by the junk pieces, may not play the original role in deriving the contextual representation of the sentence or the paragraph, even when analyzed by the powerful transformer models.

To facilitate research on language representations in the pathology domain and assist researchers in addressing the current limitations and advancing cancer research, we preset PathologyBERT, a pre-trained masked language model trained on Histopathology Specimens Reports.

We used a batch size of 32, a maximum sequence length of 64 (mean size of report is 42±26), masked language model probability of 0.15, and a learning rate of 2e-5 for pre-training the Language Model . The model was trained for 300,000 steps. All other BERT default parameters were used.

You can use this model directly with a pipeline for masked language modeling:

Refer to the original paper, Pre-trained Vs. A New Transformer Language Model for A Specific Domain - Breast Pathology Use-case for additional details and masked language performance on Pathology Specimen Reports

We also provided a Hierarchical Classification System that uses PathologyBERT as base to classify Pathology Specimen Reports.

You can test the system directly on HuggingFace: https://huggingface.co/spaces/tsantos/Hierarchical-Classification-System-for-Breast-Cancer

Github: https://github.com/thiagosantos1/HCSBC

If you have any questions, please email thiagogyn.maia@gmail.com