omicseye/

seqLens_4096_512_23M-at-xsmall

The seqLens models are a collection of genomic language models. seqLens models leverage an extensive dataset of 19,551 reference genomes, including over 18,000 prokaryotic genomes (115B nucleotides), alongside a more balanced dataset of 1,354 genomes spanning 1,166 prokaryotic and 188 eukaryotic reference genomes (180B nucleotides). Through systematic evaluation of 52 DNA language models with varying architectures, hyperparameters, and classification heads, we developed seqLens, a family of models based on disentangled attention with relative positional encoding. These models demonstrate superior performance, outperforming state-of-the-art methods in phenotypic predictions. The seqLens models provide a robust foundation for optimizing DNA language models and advancing genome annotations across diverse biological contexts.