Sparse Autoencoders for Evo 2 — BatchTopK sparse autoencoders for Arc Institute's Evo 2 genomic foundation model.
Evo 2 is a genomic foundation model capable of generalist prediction and design tasks across DNA, RNA, and proteins. It uses a frontier deep learning architecture to enable modeling of biological sequences at single-nucleotide resolution with near-linear scaling of compute and memory relative to context length. Evo 2 is trained with 40 billion parameters and 1 megabase context length on over 9 trillion nucleotides of diverse eukaryotic and prokaryotic genomes.
This repository contains the layer 26 SAE mixed prokaryote/eukaryote SAE used in the Evo 2 paper.
