Minimal HuggingFace repackage of the base variant of ModernGENA -- a ModernBERT DNA encoder pretrained on vertebrate genomes with masked language modeling.

Architecture

Parameter	Value
Parameters	136,120,832 (136.1M)
Layers	22
Attention heads	12
Embedding dimension	768
FFN hidden dimension	1152 (GeGLU; 2304-wide input/gate projection)
Vocabulary size	32768 model rows; 32000 tokenizer entries
Positional encoding	RoPE (global and local theta=10000)
Normalization	Bias-free LayerNorm (epsilon=1e-5)
Architecture	Pre-norm ModernBERT encoder with hybrid local/global attention
Local attention window	128
Global attention	Every 3 layers, starting at layer 0
Max sequence length	1024

Vocabulary: 32,000-entry GENA-LM BPE vocabulary over DNA and IUPAC symbols, including [UNK], [CLS], [SEP], [PAD], [MASK], and -. The model reserves 32,768 embedding/output rows.

Pretraining

Objective: Masked language modeling.
Data: 443 vertebrate genome assemblies totaling 353,574,093,776 bp; forward and reverse-complement strands were included.
Sampling: Regions around unique transcription start sites ([-16 kbp, +8 kbp]), with overlapping intervals merged.
Source checkpoint: AIRI-Institute/moderngena-base/model.safetensors.

Parity Verification

All 23 representation levels (embedding + 22 transformer blocks), plus the final LayerNorm output and masked-LM head, were verified to be bit-exact (max abs diff = 0.00) against the original AIRI checkpoint for matching eager, SDPA, and Flash Attention 2 backends. Verified on GPU with PyTorch 2.7.1, CUDA 12.9, transformers 4.57.6, and flash-attn 2.7.4.post1.

Related Models

See the full ModernGENA collection.

Model	Parameters	Notes
ModernGENA-base	136.1M	This model
ModernGENA-large	377.8M	Larger variant

Usage

Embedding generation

python

import torch
from transformers import AutoModel, AutoTokenizer

repo_id = "Taykhoom/ModernGENA-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    repo_id,
    trust_remote_code=True,
    attn_implementation="sdpa",
).eval()

sequences = ["ATCGATCGATCG", "GCTAGCTA"]
encoded = tokenizer(sequences, return_tensors="pt", padding=True)

with torch.no_grad():
    output = model(**encoded, output_hidden_states=True)

token_embeddings = output.last_hidden_state       # (batch, seq_len, 768)
cls_embeddings = output.last_hidden_state[:, 0]   # (batch, 768)
layer_12 = output.hidden_states[12]

For mean pooling, exclude padding with encoded["attention_mask"]. The hidden_states tuple contains the embedding output and each block output; last_hidden_state additionally applies the model's final LayerNorm.

MLM logits

python

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "Taykhoom/ModernGENA-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    attn_implementation="sdpa",
).eval()

encoded = tokenizer(["ATCGATCG"], return_tensors="pt")
encoded["input_ids"][0, 2] = tokenizer.mask_token_id
with torch.no_grad():
    logits = model(**encoded).logits  # (1, seq_len, 32768)

The pretrained MLM head is a bias-free dense projection, SiLU, bias-free LayerNorm, and a tied vocabulary decoder with bias.

Faster attention backends

python

import torch
from transformers import AutoModel

repo_id = "Taykhoom/ModernGENA-base"

# PyTorch SDPA (recommended general-purpose backend).
model = AutoModel.from_pretrained(
    repo_id,
    trust_remote_code=True,
    attn_implementation="sdpa",
)

# Flash Attention 2 (requires flash-attn and an Ampere-or-newer CUDA GPU).
model = AutoModel.from_pretrained(
    repo_id,
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
    dtype=torch.bfloat16,
)

Use attn_implementation="eager" when attention probabilities are needed. In transformers 4.57.6, SDPA falls back to eager for output_attentions=True; Flash Attention 2 does not and returns an empty attention tuple.

Fine-tuning

Standard HuggingFace conventions apply. Add a task head to AutoModel, or load AutoModelForSequenceClassification and fine-tune the newly initialized classification head with the backbone. For sequence-level tasks, use attention-mask-aware mean pooling or the [CLS] representation.

Implementation Notes

ModernGenaForMaskedLM uses the standard ModernBERT backbone and MLM head with a narrow Flash Attention 2 compatibility shim. When the MLM model requests hidden states, the shim keeps the unpadded final representation token-major until the MLM head has run, then uses ModernBERT's standard repadding. Eager, SDPA, Flash, hidden-state, logits, and loss numerics are otherwise unchanged. Load with trust_remote_code=True to enable this class.

The tokenizer strips surrounding whitespace, replaces runs of 10 or more N characters with the isolated - token, applies BPE, and wraps each sequence with [CLS] and [SEP]; it does not uppercase input. AIRI's config retains unused ModernBERT defaults bos_token_id=50281, eos_token_id=50282, and position_embedding_type="absolute". Encoding actually uses tokenizer IDs 1/2 for [CLS]/[SEP] and RoPE positions. These inert AIRI fields are preserved unchanged.

Eager attention probabilities are normalized in evaluation mode, where dropout is disabled. During training they include attention dropout, so rows are not guaranteed to sum to one.

Citation

bibtex

@article{aspidova2026_moderngena,
  title   = {Back to {BERT} in 2026: {ModernGENA} as a Strong, Efficient Baseline for {DNA} Foundation Models},
  author  = {Aspidova, Alena and Kuratov, Yuri and Shadskiy, Artem and Burtsev, Mikhail and Fishman, Veniamin},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2026.04.21.719816}
}

Credits

Original model and code by Aspidova et al. Source: AIRI checkpoint and GENA-LM. The HuggingFace repackaging was authored primarily by

License

Apache 2.0, following the original AIRI repository.