Gengram is a novel conditional memory module designed for genomic foundation models (GFMs) that introduces explicit motif memory retrieval to enhance Transformer-based DNA sequence modeling. Unlike traditional GFMs that rely on dense computation to implicitly infer multi-nucleotide motifs, Gengram provides an efficient lookup mechanism for biological patterns through a genomic-specific hashing scheme.
Figure 1 illustrates the overall architecture of Gengram, together with the evaluation pipeline used to assess its effectiveness across multiple genomic benchmarks.
Gengram
✨ Key Features
🎯 Explicit Motif Memory: Stores and retrieves k-mers (k=1-6) via hash-based lookup tables
🧬 Local Window Aggregation: 21bp window mechanism aligned with DNA helical structure
⚡ Computational Efficiency: Linear time complexity with minimal overhead
🔧 Architecture Agnostic: Compatible with various attention mechanisms (MHA, GQA, MLA)
⚖️ Stable Training: Improves load balancing in Mixture-of-Experts models
Context-dependent gating aligned with functional regions
Hierarchical representation from shallow to deep layers
2. Model Information
Model Configuration
The following details the model configuration, including the parameterization of Gengram, MoE routing strategies, and training hyperparameters used across all experiments.
Gengram Parameters
These parameters control how Gengram operates within the Transformer layers, including which layers to apply it to, the n-gram sizes, and embedding dimensions.
Parameter
Description
Example
--gengram-enabled
Enable Gengram
true
--gengram-layer-ids
Layers to apply Gengram
3 6 10
--gengram-ngram-sizes
N-gram sizes for DNA processing
1 2 3 4 5 6
--gengram-embed-dim-per-ngram
Embedding dimension per n-gram
1024
--gengram-window-size
window size
21
Mixture of Experts (MoE)
These parameters define the Mixture-of-Experts architecture, including the number of experts, routing top-k, and load balancing strategies during training.
Parameter
Description
Default
--num-experts
Number of experts
8
--moe-router-topk
Top-k experts to route to
2
--moe-router-load-balancing-type
Load balancing strategy
aux_loss
--moe-aux-loss-coeff
Auxiliary loss coefficient
1e-3
Training Parameters
These parameters specify the training setup, including sequence length, batch sizes, precision, and attention optimizations.
Gengram demonstrates strong performance across multiple genomic benchmarks, achieving competitive results despite being trained on significantly fewer tokens and with a smaller model size.
This repository and the Gengram model weights are licensed under the Apache License 2.0.
Please note that the primary use of Gengram model is to support genomics research, providing researchers with advanced analytical capabilities and long-context modeling tools powered by large-scale foundation models for the human genome. It is not intended for use in any manner that violates applicable laws or regulations, nor for any activities prohibited by the license agreement.
6. Citation and Acknowledgements
We acknowledge the high-quality sequencing data provided by CycloneSEQ, which forms an important foundation for this work. We also appreciate the inspiration from DeepSeek's Engram module and the framework support provided by Megatron-LM. Model training was conducted on the 021 Science Foundation Model and Zero2X open platform.
If you use this work in your research, please cite the following paper:
bibtex
1@article@article{gengram2026,
2 title={Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram},
3 author={Genos Team and Xu, Huinan and Feng, Xuyang and Chen, Junhong and Liu Junchen and Deng, Kaiwen and Ding, Kai and Long, Shengning and Shuai, Jiaxue and Li, Zhaorong and Liu, Shiping and Xue, Guirong and Xiao, Zhan},
4 journal={arXiv preprint arXiv:2601.22203},
5 year={2026}
6}
7. Contact
For project-related questions, please open an issue. You can also contact the Genos Team at Genos@genomics.cn.
