recommended maximum sequence length for practical inference: up to 1,000,000 nucleotides
The architecture combines three components:
GENA-LM backbone (BERT Large)
A pretrained genomic language model that produces contextual sequence representations.
Recurrent Memory Transformer (RMT)
Long sequences are processed in segments with memory tokens, allowing the model to operate beyond the base backbone context length.
1D UNET segmentation head
Token-level representations are repeated to nucleotide resolution and refined with repeated 1D convolutional UNET passes before final projection to the five output classes.
Batch Size Limitation
The current implementation supports batch size = 1 only.
This limitation comes from the 1D convolutional UNET refinement head used after token repetition to nucleotide resolution.
Inference is therefore currently performed one sequence at a time.
Context Length
This model is not limited to a fixed short context window in the usual backbone sense because it uses RMT-based segmented processing with memory tokens.
In practice, we currently recommend using sequences of no more than 1,000,000 nucleotides per inference run.
Training Data
This model was fine-tuned on gene sequences only, not on full genomes.
Training data includes:
mRNA transcripts
lncRNA transcripts
Dataset characteristics:
one transcript per gene
no intergenic regions
multispecies training dataset
Each training sample corresponds to a single gene sequence.
