GwasGPT
Overview
The GwasGPT model was proposed in GwasGPT: generative pre-trained transformer for statistical genomic text generation and mining by Michael Chase.
GwasGPT is a domain-specific generative pre-trained Transformer language model for statistical genomic text generation and mining.
GwasGPT follows the Transformer language model backbone, and is pre-trained on 15M PubMed abstracts from scratch.
The abstract from the paper is the following:
Pre-trained language models have attracted increasing attention in the statistical genetics domain, inspired by their great success in the general natural language domain.
Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the statistical genomics domain, such as BioBERT, BioGPT, and PubMedBERT.
While they have achieved great success on a variety of discriminative downstream genomic tasks, the lack of generation ability constrains their application scope. In this paper, we propose GwasGPT, a domain-specific generative Transformer language model pre-trained on large-scale genomic literature.
We evaluate GwasGPT on six genomic natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record.
Our case study on text generation further demonstrates the advantage of GwasGPT on GWAS (Genome-Wide-Association-Studies) literature to generate fluent descriptions for genomic terms.
Tips:
GwasGPT is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather than the left.
GwasGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows GwasGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script.
The model can take the past_key_values (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage.
