Views
No views yet
| Parameter | Value |
|---|---|
| Layers | 12 |
| Attention heads | 12 |
| Embedding dimension | 768 |
| FFN hidden dimension | 3072 (GELU) |
| Vocabulary size | 32000 |
| Positional encoding | learned absolute |
| Normalization | Pre-LayerNorm (eps=1e-12); final-layer LayerNorm: No |
| Architecture | Pre-LayerNorm BERT (without a final-layer LayerNorm) |
| Max sequence length | 512 BPE tokens (~4608 nucleotides) |
[CLS], [SEP], [PAD],
[UNK], and [MASK].AIRI-Institute/gena-lm-bert-base-t2t-multiAIRI-Institute/gena-lm-bert-base-t2t-multi weights, for the eager backend. The added sdpa and
flash_attention_2 backends agree with eager up to the expected fused-kernel
floating-point tolerance. Verified on GPU with PyTorch 2.7 / CUDA 12.9.| Model | Parameters | Notes |
|---|---|---|
| GENA-LM-bert-base | 110M | 12L / 768d, 512 ctx |
| GENA-LM-t2t-bert-base | 110M | 12L / 768d, 512 ctx |
| GENA-LM-t2t-multi-species-bert-base | 110M | 12L / 768d, 512 ctx (this model) |
| GENA-LM-t2t-lastln-base | 110M | 12L / 768d, 512 ctx |
| GENA-LM-t2t-bert-large | 336M | 24L / 1024d, 512 ctx |
| GENA-LM-t2t-bigbird-base | 110M | 12L / 768d, 4096 ctx |
| GENA-LM-t2t-sparse-bigbird-base | 110M | 12L / 768d, 4096 ctx |
| GENA-LM-sparse-bigbird-base | 110M | 12L / 768d, 4096 ctx |
1import torch
2from transformers import AutoTokenizer, AutoModel
3
4tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True)
5model = AutoModel.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True)
6model.eval()
7
8sequences = ["ACGTACGTACGTACGT", "TTACGGGCATACGACGT"]
9enc = tokenizer(sequences, return_tensors="pt", padding=True)
10
11with torch.no_grad():
12 out = model(**enc)
13
14cls_emb = out.last_hidden_state[:, 0, :] # (batch, dim) -- CLS token
15token_emb = out.last_hidden_state # (batch, seq_len, dim)
16
17# Intermediate layers
18out_all = model(**enc, output_hidden_states=True)
19layer6_emb = out_all.hidden_states[6]1from transformers import AutoTokenizer, AutoModelForMaskedLM
2
3tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True)
4model = AutoModelForMaskedLM.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True)
5model.eval()
6
7enc = tokenizer(["ACGT[MASK]CGTACGT"], return_tensors="pt")
8with torch.no_grad():
9 logits = model(**enc).logits # (1, seq_len, vocab_size)1# SDPA (PyTorch 2.0+) -- recommended for production
2model = AutoModel.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True,
3 attn_implementation="sdpa")
4
5# Flash Attention 2 (requires flash-attn) -- fastest on long sequences
6import torch
7model = AutoModel.from_pretrained("Taykhoom/GENA-LM-t2t-multi-species-bert-base", trust_remote_code=True,
8 attn_implementation="flash_attention_2",
9 dtype=torch.bfloat16)[CLS] token embedding as input to a prediction head.sdpa and flash_attention_2 attention backends, selectable via attn_implementation;
the eager backend reproduces the original outputs bit-for-bit. The original NSP head
and pooler are not included, since this port targets embedding and masked-LM use.
AutoModel returns the backbone without a pooler; use the [CLS] hidden state or
masked mean pooling for sequence embeddings. The input embeddings and MLM decoder are tied.1@article{fishman2025_genalm,
2 title = {{GENA-LM}: a family of open-source foundational {DNA} language models for long sequences},
3 author = {Fishman, Veniamin and Kuratov, Yuri and Shmelev, Aleksei and Petrov, Maxim and Penzar, Dmitry and Shepelin, Denis and Chekanov, Nikolay and Kardymon, Olga and Burtsev, Mikhail},
4 journal = {Nucleic Acids Research},
5 volume = {53},
6 number = {2},
7 pages = {gkae1310},
8 year = {2025},
9 doi = {10.1093/nar/gkae1310}
10}