The plant DNA large language models (LLMs) contain a series of foundation models based on different model architectures, which are pre-trained on various plant reference genomes.
All the models have a comparable model size between 90 MB and 150 MB, BPE tokenizer is used for tokenization and 8000 tokens are included in the vocabulary.
The model is trained based on the OpenAI GPT-2 model with modified tokenizer specific for DNA sequence.
This model is fine-tuned for predicting H3K27ac histone modification.
1from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
2
3model_name = 'plant-dnagpt-BPE-H3K27ac'
4# load model and tokenizer
5model = AutoModelForSequenceClassification.from_pretrained(f'zhangtaolab/{model_name}', trust_remote_code=True)
6tokenizer = AutoTokenizer.from_pretrained(f'zhangtaolab/{model_name}', trust_remote_code=True)
7
8# inference
9sequences = ['GCTTTGGTTTATACCTTACACAACATAAATCACATAGTTAATCCCTAATCGTCTTTGATTCTCAATGTTTTGTTCATTTTTACCATGAACATCATCTGATTGATAAGTGCATAGAGAATTAACGGCTTACACTTTACACTTGCATAGATGATTCCTAAGTATGTCCT',
10 'TAGCCCCCTCCTCTCTTTATATAGTGCAATCTAATATATGAAAGGTTCGGTGATGGGGCCAATAAGTGTATTTAGGCTAGGCCTTCATGGGCCAAGCCCAAAAGTTTCTCAACACTCCCCCTTGAGCACTCACCGCGTAATGTCCATGCCTCGTCAAAACTCCATAAAAACCCAGTG']
11pipe = pipeline('text-classification', model=model, tokenizer=tokenizer,
12 trust_remote_code=True, top_k=None)
13results = pipe(sequences)
14print(results)
15
We use GPT2ForSequenceClassification to fine-tune the model.
Detailed training procedure can be found in our manuscript.
Model was trained on a NVIDIA GTX1080Ti GPU (11 GB).
