Views
No views yet
| Parameter | Value |
|---|---|
| Layers | 6 |
| Attention heads | 8 |
| Embedding dimension | 128 |
| FFN hidden dimension | 320 (SwiGLU) |
| Vocabulary size | 16 |
| Positional encoding | RoPE (rotary_percent=1.0) |
| Normalization | RMSNorm |
| MLP activation | SwiGLU |
| Architecture | Pre-LN Transformer |
| Max sequence length | 1024 (training truncation; RoPE has no hard limit) |
[PAD], [MASK], [CLS], [SEP], [UNK], A, G, C, T, U, N,
[BOS], [EOS], [UNUSED1], [UNUSED2], [UNUSED3]A, C, G, T), as in the upstream implementation.
Convert U to T before tokenization; the tokenizer treats them as distinct vocabulary entries.genbio-ai/AIDO.RNA-1M-MARSgenbio-ai/AIDO.RNA-1M-MARS weights at all
7 representation levels (embedding + 6 transformer layers).
Intermediate layer differences are floating-point accumulation noise normalised away by the
final layer norm; the final output agrees under torch.allclose(atol=1e-5, rtol=1e-5).
max diff = 1.65e-05 (final output), 2.57e-05 (intermediate layers). Verified on GPU with PyTorch 2.7 / CUDA 12.| Model | Parameters | Data | Notes |
|---|---|---|---|
| AIDO.RNA-1M-MARS | 1M | MARS ncRNA | Smallest MARS variant |
| AIDO.RNA-25M-MARS | 25M | MARS ncRNA | Mid-size MARS variant |
| AIDO.RNA-300M-MARS | 300M | MARS ncRNA | Large MARS variant |
| AIDO.RNA-650M | 650M | 42M ncRNA | Base model |
| AIDO.RNA-650M-CDS | 650M | 42M ncRNA + CDS | CDS-adapted |
| AIDO.RNA-1.6B | 1.6B | 42M ncRNA | Largest base model |
| AIDO.RNA-1.6B-CDS | 1.6B | 42M ncRNA + CDS | Largest CDS-adapted |
1import torch
2from transformers import AutoTokenizer, AutoModel
3
4tokenizer = AutoTokenizer.from_pretrained("Taykhoom/AIDO.RNA-1M-MARS", trust_remote_code=True)
5model = AutoModel.from_pretrained("Taykhoom/AIDO.RNA-1M-MARS", trust_remote_code=True)
6model.eval()
7
8sequences = ["ACGTGCTAGCTAGCTA", "ATGCTAGCTAGCTAGC"]
9enc = tokenizer(sequences, return_tensors="pt", padding=True)
10
11with torch.no_grad():
12 out = model(**enc)
13
14cls_emb = out.last_hidden_state[:, 0, :] # (batch, 128) -- CLS token
15token_emb = out.last_hidden_state # (batch, seq_len, 128)
16
17# Intermediate layers
18out_all = model(**enc, output_hidden_states=True)
19layer3_emb = out_all.hidden_states[3]1from transformers import AutoTokenizer, AutoModelForMaskedLM
2
3tokenizer = AutoTokenizer.from_pretrained("Taykhoom/AIDO.RNA-1M-MARS", trust_remote_code=True)
4model = AutoModelForMaskedLM.from_pretrained("Taykhoom/AIDO.RNA-1M-MARS", trust_remote_code=True)
5model.eval()
6
7enc = tokenizer(["ACGT[MASK]GCTA"], return_tensors="pt")
8with torch.no_grad():
9 logits = model(**enc).logits # (1, seq_len, 16)cls_emb = out.last_hidden_state[:, 0, :] (CLS token) as
input to a task-specific head for sequence-level tasks.genbio-ai/AIDO.RNA-1M-MARS checkpoint requires the
ModelGenerator package to load.
This port is a clean standalone re-implementation:modeling_aidorna.py and configuration_aidorna.py.U/u to T/t, matching the original checkpoint preprocessing.attn_implementation="sdpa" and attn_implementation="flash_attention_2" are added
(not present in the original genbio-ai implementation).1@article{zou2024_aidorna,
2 title = {A Large-Scale Foundation Model for {RNA} Function and Structure Prediction},
3 author = {Zou, Shuxian and Tao, Tianhua and Mahbub, Sazan and Ellington, Caleb N. and Algayres, Robin and Li, Dian and Zhuang, Yonghao and Wang, Hongyi and Song, Le and Xing, Eric P.},
4 journal = {bioRxiv},
5 year = {2024},
6 doi = {10.1101/2024.11.28.625345}
7}LICENSE for details.