GenerRNA is a generative pre-trained language model for de novo RNA sequence design. It is a Transformer (decoder-only, GPT-style) model that learns the "language" of RNA from millions of natural sequences and can generate novel, realistic RNA sequences without any structural input, functional label, or sequence alignment. To our knowledge, GenerRNA is the first application of a generative language model to RNA generation.
With GenerRNA you can:
Generate RNA in a zero-shot manner to explore the RNA sequence space, or
Fine-tune on your own dataset to generate RNAs belonging to a particular family or possessing specific characteristics (e.g., high binding affinity to a target protein).
GenerRNA is a Transformer decoder-only (GPT-style) language model trained on RNA nucleotide sequences. By treating RNA as a sequence of tokens, it learns statistical and structural regularities of RNA directly from data and can then sample entirely new sequences. GenerRNA was pre-trained on ~16 million RNA sequences (16.09M), encompassing ~17.4 billion nucleotides. Generated RNAs are novel (distinct from training sequences) yet fold into stable secondary structures, and the model can be fine-tuned to design functional RNAs such as protein binders — all without requiring prior structural knowledge.
Key Features
🧬 De novo RNA generation — create novel RNA sequences from scratch; no structure, label, or alignment required.
🎯 Zero-shot or fine-tuned — explore RNA space out of the box, or specialize the model for a target family or function.
Preferred Networks, Inc. & The University of Tokyo
Intended Use & Use Cases
GenerRNA is intended for research in RNA biology, synthetic biology, and RNA-based therapeutics / drug discovery. Typical use cases include:
Exploring the diversity of the RNA sequence space.
Generating candidate RNAs from a target family by fine-tuning on family-specific data.
Designing RNAs with desired functional properties, such as aptamers/binders with high affinity to a target protein (demonstrated for the RNA-binding proteins ELAVL1 and SRSF1 in the paper).
Serving as a pre-trained backbone for downstream RNA modeling and design tasks.
Requirements
A CUDA environment with a minimum of 8 GB VRAM is required.
.
├── LICENSE
├── README.md
├── CITATION.cff # machine-readable citation metadata
├── model.py # model architecture (decoder-only Transformer)
├── sampling.py # generate sequences from a trained model
├── tokenization.py # tokenize sequence data for training
├── train.py # pre-training / fine-tuning entry point
├── train_BPE.py # train a new BPE tokenizer
├── model_updated.pt # recommended checkpoint (longer context, deduplicated data)
├── tokenizer/ # BPE tokenizer for model_updated.pt
├── configs/
│ ├── example_pretraining.py
│ └── example_finetuning.py
└── experiment_data/
├── *.csv # data underlying the paper's figures
├── pretraining_data.sh # how the pre-training corpus was built (RNAcentral + MMseqs2)
└── historical_version/ # original model (split into parts) + its tokenizer
├── model.pt.part-a{a,b,c,d}
└── tokenizer_bpe_1024/
Training Data
GenerRNA was pre-trained on RNA sequences from RNAcentral (release 22, which aggregates 51 expert databases). Starting from 34.39 million raw sequences, deduplication with MMseqs2 at 80% sequence identity yielded a pre-training corpus of ~16 million sequences (16.09M), encompassing ~17.4 billion nucleotides. GenerRNA has a context window of 1024 tokens (~4000 nucleotides). The pre-processing pipeline is in experiment_data/pretraining_data.sh, and the data underlying the paper's figures is provided in experiment_data/. See the paper for full dataset details.
Limitations
GenerRNA models RNA sequence; it does not explicitly predict tertiary structure or function. Validate candidates with downstream structure/function tools and wet-lab experiments.
A CUDA GPU is required for generation and training as provided.
Zero-shot outputs reflect the natural distribution of the training data; targeting a specific family or property generally requires fine-tuning.
Generated sequences are computational hypotheses and should be experimentally validated before any real-world application.
FAQ
What is GenerRNA?
GenerRNA is a generative, pre-trained language model (a decoder-only Transformer) that designs novel RNA sequences de novo, without requiring structural information, functional labels, or sequence alignments.
How is GenerRNA different from other RNA models?
Most RNA models are discriminative — they predict structure or properties from a given sequence. GenerRNA is generative: it samples entirely new sequences. To our knowledge, it is the first application of a generative language model to RNA generation.
Do I need RNA structure or alignments as input?
No. GenerRNA generates sequences directly from its learned distribution; no structure or alignment is needed.
Can I generate RNAs from a specific family or with a specific function?
Yes. Fine-tune GenerRNA on a family- or function-specific dataset. The paper demonstrates designing RNAs with high binding affinity to the proteins ELAVL1 and SRSF1.
Which checkpoint should I use?
Use model_updated.pt (longer context, trained on deduplicated data). The original split model is kept in experiment_data/historical_version/ for reproducibility.
Is GenerRNA free to use?
Yes. The code and weights are released under the MIT License. Please cite the paper if you use GenerRNA in your work.
If you use GenerRNA, its checkpoints, or this repository in your research, please cite:
bibtex
1@article{zhao2024generrna,
2 title = {GenerRNA: A generative pre-trained language model for de novo RNA design},
3 author = {Zhao, Yichong and Oono, Kenta and Takizawa, Hiroki and Kotera, Masaaki},
4 journal = {PLOS ONE},
5 volume = {19},
6 number = {10},
7 pages = {e0310814},
8 year = {2024},
9 doi = {10.1371/journal.pone.0310814},
10 publisher = {Public Library of Science}
11}
Plain text: Zhao Y, Oono K, Takizawa H, Kotera M (2024) GenerRNA: A generative pre-trained language model for de novo RNA design. PLOS ONE 19(10): e0310814. https://doi.org/10.1371/journal.pone.0310814
