This is a distilled version of
DNABERT by using DistilBERT technique. It has a BERT architecture with 6 layers and 768 hidden units, pre-trained on 6-mer DNA sequences. For more details on the pre-training scheme and methods, please check the original
thesis report.
The model can be used to fine-tune on a downstream genomic task, e.g. promoter identification.
1import torch
2from transformers import DistilBertForSequenceClassification
3
4model = DistilBertForSequenceClassification.from_pretrained('Peltarion/dnabert-distilbert')
More details on how to fine-tune the model, dataset and additional source codes are available on
github.com/joanaapa/Distillation-DNABERT-Promoter.
